Iterative media object compression algorithm optimization using decoupled calibration of perceptual quality algorithms

ABSTRACT

One or more multi-stage optimization iterations are performed with respect to a compression algorithm. A given iteration comprises a first stage in which hyper-parameters of a perceptual quality algorithm are tuned independently of the compression algorithm. A second stage of the iteration comprises tuning hyper-parameters of the compression algorithm using a set of perceptual quality scores generated by the tuned perceptual quality algorithm. The final stage of the iteration comprises performing a compression quality evaluation test on the tuned compression algorithm.

BACKGROUND

Many companies and other organizations provide access to their servicesand products to customers over networks such as the public Internet. Insome cases, substantial amounts of media content, such as still images,video or audio may be transferred by such entities to their end users.For example, content providers may enable users to download electronicversions of books and magazines, listen to downloaded or streamingversions of music and films, and so on. E-retail enterprises may enableclients to view images of products before making purchasing decisions,while real-estate agents, renters, travel services and the like mayprovide photographs and/or videos of accommodations to their endcustomers. An Internet-scale e-retail enterprise may provide millions oreven billions of images to clients every day.

The quality of the images, videos, or audios, as perceived by the endusers, may sometimes have a significant impact on whether the end usersdecide to engage further with the organization presenting the data:e.g., whether a given product is purchased from an e-retail web site,whether a given hotel room or bed-and-breakfast is reserved by avacationer, and so on. Thus, from the service providing organization'sperspective, providing media that looks good (and in cases where audiois involved, sounds good) may be critical to the success of the servicebeing provided.

To provide the best possible experience for the end user, uncompressedor original versions of the available media objects (image files, audiofiles, video files, etc.) could of course be presented, at least intheory. However, this approach may be impractical in several ways.First, it may consume very high amounts of network bandwidth. Second,the delays experienced by users between the times that they request amedia object and the time that the media object is displayed may becomeexcessive. In order to avoid such problems, compression techniques aretypically used: the media objects are compressed at the sendingorganization (which can involve the loss of some information from theoriginal objects), transferred in compressed form over the network, andde-compressed at the receiving device before being presented to the endcustomer.

Because of the loss of information involved in the process, the versionof the media object presented to the end customer may typically besomewhat degraded relative to the original: that is, if an end customerwere given the opportunity to compare the original versus the presentedversion, the end customer may at least in some cases consider thepresented version to be somewhat worse than the original. The extent ofthe perceived degradation depends on various factors, including theparticular compression technique used. Unfortunately, quantifiablephysical differences between the original and compressed versions of themedia objects are sometimes insufficient to accurately predict perceiveddegradation in quality. Numerous compression algorithms, some with verylarge numbers of tunable hyper-parameters, have been developed over theyears for various kinds of media. Predicting the extent of the perceiveddegradation for various compression schemes, and using such predictionsto tune compression algorithms for large-scale presentation of mediaobjects, remain non-trivial technical challenges.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an example system environment in which anetwork-accessible service for optimizing the process of compressingmedia content for over-the-network presentation may be implemented,according to at least some embodiments.

FIG. 2 illustrates example factors which may influence the perceivedquality of a compressed image, according to at least some embodiments.

FIG. 3 illustrates example variations in the relationship between imagecompression ratios and perceived quality of the images, according to atleast some embodiments.

FIG. 4 illustrates example inputs and outputs of a perceived qualitydegradation estimation algorithm which may be used to select compressedimages for presentation to end users, according to at least someembodiments.

FIG. 5 illustrates example variations in quality scores produced bydifferent perceptual quality algorithms, according to at least someembodiments.

FIG. 6 illustrates an example technique for preparing a training dataset for a machine learning model for predicting perceived image quality,according to at least some embodiments.

FIG. 7 illustrates an example architecture of a neural network-basedmodel for predicting perceived image quality based on scores generatedby multiple other perceptual quality algorithms, according to at leastsome embodiments.

FIG. 8 is a flow diagram illustrating aspects of operations which may beperformed to predict perceived image quality using a fusion model whoseinput comprises respective scores from multiple other perceptual qualityalgorithms on a combination of adversarial and non-adversarial trainingexamples, according to at least some embodiments.

FIG. 9 is a flow diagram illustrating aspects of operations that may beperformed in a two-stage process to train a machine learning model forperceived image quality prediction, in which adversarial trainingexamples are used in one of the stages and non-adversarial trainingexamples are used in the other stage, according to at least someembodiments.

FIG. 10 illustrates example improvements in results which may beachieved using a fusion model for predicting perceived image quality,according to at least some embodiments.

FIG. 11 illustrates example programmatic interactions associated withthe training of a fusion model for predicting perceived image quality,according to at least some embodiments.

FIG. 12 illustrates example compressed image file formats andcompression algorithm sub-operations, according to at least someembodiments.

FIG. 13 is a flow diagram illustrating a high-level overview of aniterative compression algorithm optimization technique, according to atleast some embodiments.

FIG. 14 is a flow diagram illustrating a high-level overview of anevolutionary algorithm which may be employed at one or more stages of aniterative compression algorithm optimization technique, according to atleast some embodiments.

FIG. 15 is a flow diagram illustrating aspects of an iterativecompression algorithm optimization technique in which evolutionarytechniques are employed, according to at least some embodiments.

FIG. 16 illustrates example programmatic interactions between a clientand a media optimization service, pertaining to iterative compressionalgorithm optimization, according to at least some embodiments.

FIG. 17 is a flow diagram illustrating aspects of a technique in whichimage class-specific tuning of compression algorithms may be conducted,according to at least some embodiments.

FIG. 18 is a flow diagram illustrating aspects of a technique in whichcustom tuning of compression algorithms may be conducted for sub-imagesof a given image, according to at least some embodiments.

FIG. 19 is a block diagram illustrating an example computing device thatmay be used in at least some embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include,” “including,” and“includes” mean including, but not limited to. When used in the claims,the term “or” is used as an inclusive or and not as an exclusive or. Forexample, the phrase “at least one of x, y, or z” means any one of x, y,and z, as well as any combination thereof.

DETAILED DESCRIPTION

The present disclosure relates to methods and apparatus for enhancedprediction of perceived quality of compressed media objects such asimages, as well as efficient iterative techniques for optimizingcompression algorithms used for presenting such media objects to mediaconsumers. According to at least some embodiments, a network-accessibleservice referred to as a media optimization service (MOS) may be set upat a provider network or cloud computing environment to implement suchtechniques on behalf of clients of the service.

At a high level, one of the proposed techniques comprises theintelligent selection of examples (e.g., image sets which meet certaincriteria with respect to the divergence of quality scores obtained usingpre-existing quality degradation prediction techniques) for training arelatively simple machine learning model which uses multiple qualityscores as input for a given training example, with each of the scoresbeing obtained from a respective existing perceived quality degradationalgorithm. The trained machine learning model may, in effect, combinethe strengths of each of the existing algorithms used for its input,while avoiding some of the weaknesses of such existing algorithms bylearning about subtle perceived quality differences or “edge cases” as aresult of the intelligent selection of the training examples. After themodel is trained, it may predict perceived image quality degradationsmore accurately, and in a more generalizable way, for variouscompression algorithms and hyper-parameter combinations than thepre-existing techniques. The terminology “PQ algorithm” (perceptualquality algorithms or perceived quality algorithms) may be used hereingenerically to refer to algorithms that predict perceived media qualitydegradations, e.g., by generating numerical “PQ scores” indicative ofthe predicted extent of the human-perceived degradation of a givencompressed media object relative to a reference or baseline version ofthe same media object, or the absolute perceived quality of a mediaobject where the reference version is unknown. Generally speaking, PQalgorithms may be used for a variety of types of media (e.g., audio andvideo in addition to still images) in various embodiments. The referenceversion may be an uncompressed version of the object in some cases, or adifferent compressed version generated using a default or referencecompression algorithm, depending on the scenario in which the PQalgorithm is being used in various embodiments. PQ algorithms may bereferred to as PQ “metrics” and/or as “perceived quality” algorithms insome embodiments, and reference versions may also be referred to as“original” or “baseline” versions in some embodiments. In theabove-described methodology, a new PQ algorithm which consumes or“fuses” scores generated by other PQ algorithms (for specially-selectedtraining examples) as input, and does not have to be provided the actualmedia objects being compared as input, is trained and deployed invarious embodiments. As such, the new PQ algorithm may be referred to asa “fusion” algorithm in at least some embodiments, and the underlyingmachine learning model may be referred to as a “fusion” model. Resultsobtained using the fusion PQ algorithm may be used, for example, to makebetter tradeoffs with regard to compression approaches (includingtradeoffs between file sizes and specific hyper-parameter combinationsof the compression algorithms while meeting quality requirements) to beemployed for presenting content items to content consumers in variousembodiments, and/or to select particular compressed images to bepresented to the content consumers from among a set of availablecompressed images.

A second proposed technique utilizes an iterative evolutionary approachtowards selecting optimized values of hyper-parameters for imagecompression algorithms in at least some embodiments. This secondtechnique resolves the “chicken-and-egg” problem confronting attempts totune compression algorithms: namely, that in order to tune a compressionalgorithm in an automated manner, one usually needs a sufficiently-tunedPQ algorithm (e.g., to judge the merits of respective compressedversions of a given image), but the process of tuning the PQ algorithmusing traditional approaches may itself require using asufficiently-tuned compression algorithm (e.g., to enable usefulcomparisons of different combinations of PQ algorithm hyper-parametersrelative to one another). In the proposed approach, a number ofoptimization iterations, each involving several stages, may be run invarious embodiments. In a first stage of a given optimization iteration,a PQ algorithm is first calibrated or tuned independently of anyparticular combination of compression hyper-parameters, e.g., using datasets obtained from human annotators and a selected optimizationstrategy, such as an evolutionary methodology based on a first set oftradeoffs between exploration and exploitation and a first fitnessfunction. Then, in a second stage of the same optimization iteration,the tuned PQ algorithm is used to generate PQ scores for images producedusing a variety of compression hyper-parameter combinations. Anotheroptimization strategy (e.g., another evolutionary methodology which usesa different set of exploration-versus-exploitation tradeoffs and adifferent fitness function) in which such scores are compared with oneanother may then be used to select better-performing compressionalgorithm hyper-parameter combinations in various embodiments. In atleast some embodiments, a third stage of a given optimization iterationmay involve evaluating the “fittest” of the hyper-parameter combinationsidentified for the compression algorithm using human annotators and newimage sets which have not been used thus far in any of the iterations,along with statistical techniques which address potential biases in theevaluation. If the result of the evaluation stage suggest thatadditional iterations are to be performed, some of the annotationsproduced in the evaluation stage may be used in the PQ algorithm tuningstage of the next iteration in at least some embodiments. Tuned versionsof the compression algorithms obtained using the iterative multi-stageapproach outlined above may be deployed for content presentation invarious embodiments, providing compressed versions of the images whichmeet selected quality criteria while ensuring that the file sizes of thecompressed versions remain within reasonable limits.

As one skilled in the art will appreciate in light of this disclosure,certain embodiments may be capable of achieving various advantages,including some or all of the following: (a) reducing the overall amountof network, computing, and/or storage resources which have to be used toprovide media objects such as images to consumers such as e-retailwebsite customers, e-book readers, etc., and/or (b) improving the userexperience of such media consumers, e.g., by reducing the latencies ordelays experienced by the consumers for content item presentation.

According to some embodiments, a system may comprise one or morecomputing devices. The computing devices may include instructions thatupon execution on or across one or more processors of the computingdevices cause the computing devices to identify, for inclusion in agroup of images used for training a machine learning model, a firstplurality of image tuples which satisfy an algorithm-to-algorithmdivergence threshold. Individual image tuples may comprise a referenceimage, a first compressed version of the reference image, and a secondcompressed version of the reference image. The image tuples may beselected or filtered from an available larger collection of image tuplesin various embodiments. The image tuples may be selected/filtered suchthat a difference between (a) a first quality degradation score producedby a first PQ algorithm of a first set of PQ algorithms for one or moreof the compressed versions relative to the reference image, and (b) asecond quality degradation score produced by a second PQ algorithm ofthe first set for the one or more of the compressed versions relative tothe reference image exceeds the divergence threshold. PQ algorithms maydiffer in their respective judgements/predictions of quality degradationfor the same image pair, for example, because the underlying assumptionsof the PQ algorithms regarding the important features of images maydiffer, because their underlying assumptions about the human visualsystem may differ, and so on. The intuition behind usingalgorithm-to-algorithm divergence as a factor to select trainingexamples is that such divergent images, when evaluated for qualitydegradations by human annotators, may provide examples of subtledifferences in quality that would be valuable for the machine learningmodel to learn, and that such subtle differences may not be capturedsufficiently by the existing set of disagreeing PQ algorithms.

It may be helpful at this stage to consider a concrete example ofalgorithm-to-algorithm divergence. Consider a simplified scenario inwhich a PQ score is a real number between 0 and 1, with 0 representingno human-perceived degradation (as predicted by a PQ algorithm) and 1representing a very high level of human-perceived degradation. Considertwo image tuples: tuples T1 and T2. T1 comprises a reference image I1and compressed versions C1′I1 and C2′I1 of I1. T2 comprises a referenceimage 12 and compressed versions C1′I2 and C2′I2 of I2. Assume furtherthat a set of PQ algorithms PQ1 and PQ2 are used to identify imagetuples which satisfy a divergence threshold. PQ1 generates a PQ score of0.2 for the image pair (I1, C1′I1) of T1 and a PQ score of 0.3 for theimage pair (I1, C2′I1) of T1. PQ2 generates a PQ score of 0.22 for theimage pair (I1, C1′I1) of T1 and a PQ score of 0.27 for the image pair(I1, C2′I1) of T1. Based on these example scores, the mean absolutedifference between the scores produced by the two algorithms PQ1 and PQ2for the image pairs of T1 is quite small: 0.015, the mean of|0.2−0.22|=0.02 for (IL C1′I1), and |0.3−0.27|=0.03 for (I1, C2′I1).Assume that PQ1 generates a PQ score of 0.4 for the image pair (I2,C1′I2) of T1 and a PQ score of 0.7 for the image pair (I2, C2′I2) of T2while PQ2 generates a PQ score of 0.2 for the image pair (I2, C1′I2) ofT2 and a PQ score of 0.25 for the image pair (I2, C2′I2) of T2. The meanabsolute difference between the scores produced by the two algorithmsPQ1 and PQ2 for the image pairs of T2 is larger than for the image pairsof T1: 0.325, the mean of |0.4−0.2|=0.2 for (I2, C1′I2), and|0.7−0.25|=0.45 for (I1, C2′I1). If the divergence criteria forselecting image tuples is based on such mean absolute differencemeasures, and the divergence threshold is set to a value such as 0.25,then T1 would not be selected (as the divergence 0.015 in its scoreswould not be as high as the threshold 0.25), while T2 would be selected(as the divergence 0.325 in its scores is higher than the threshold0.25). Note that in practice, other measures than mean absolutedifferences may be used in at least some embodiments, and/or the numberof PQ algorithms whose scores are compared with one another may behigher than two. In some cases, for example, the PQ score divergence fora single compressed image of a tuple relative to the reference image ofthe tuple may be sufficient to designate an image tuple as meeting thealgorithm-to-algorithm divergence criteria, even if the PQ scores forthe other image of the tuple do not vary much from one algorithm toanother. The image tuples or pairs that are identified using suchcriteria may be referred to as “adversarial” images in some embodiments,as they represent examples of disagreement among existing PQ algorithms.The set of existing PQ algorithms used to identify adversarial imagesmay be chosen based on a variety of factors in different embodiments,e.g., based on a knowledge base which contains entries indicating whichexisting PQ algorithms have proven individually successful for variousimage presentation scenarios in the past, based on input from clients ofa media optimization service, based on relative resource usage metricsof the PQ algorithms, and so on.

After the adversarial image tuples are identified using a first selectedset of PQ algorithms, respective labels may be obtained for theadversarial image tuples from a group of one or more annotators invarious embodiments. A label for a given image tuple may, for example,indicate which compressed version of the given image tuple is perceivedto be more similar to the reference image of the given image tuple bythe human annotators. This process of, in effect, resolving thedifferences between the disagreeing PQ algorithms using human judgementsmay be referred to as “adversarial” annotation in some embodiments.

In at least some embodiments, another group of image tuples selectedfrom the image collection may be automatically annotated for trainingthe machine learning model. Such auto-labeling may be performed using PQscores generated for the image tuples by a second set of PQ algorithms,instead of utilizing human annotators. This second group of image tuplesmay be referred to as “non-adversarial” images in at least someembodiments, as they may be selected for reasons other than divergencebetween PQ algorithms' scores. Such reasons may, for example, include(a) that the PQ scores generated for a given image tuple by therespective PQ algorithms of the second set meet a high level ofconcordance or agreement among themselves, and/or (b) that thecompressed images of the tuples represent common or well-known examplesof compression-related artifacts (e.g., introduced spurious edges,reduction of contrast, etc.) which are expected to be noticeable tohumans. The intuition behind identifying the non-adversarial images isthat such images, when use to generate training records for the machinelearning model, may provide examples of more obvious differences inquality that would be obtained from the adversarial images alone, andmay thus strengthen the ability of the model to respond tostraightforward quality degradation scenarios (and not just to respondto subtle differences learned from the adversarial examples).

The two sets of labeled image tuples (adversarial and non-adversarial)may be combined into a single labeled image data set in someembodiments. However, as discussed below, in at least one embodiment thetwo sets may be used separately for respective stages of a multi-stagetraining procedure used for the machine learning model.

Using a third set of PQ algorithms, a plurality of pairs of trainingrecords may be generated for the machine learning model from the labeledimage data set in some embodiments. A given pair of training records maycomprise a first record and a second record, each comprising multiple PQscores for an image pair comprising one of the compressed versions of animage tuple. The first record may include, in at least some embodiments,(a) a plurality of quality degradation scores for a first compressedversion of a particular reference image of the labeled image data set,obtained using respective PQ algorithms of the third set, and (b) theparticular label which was stored in the labeled image data set for theimage tuple of which the first compressed version is a member.Similarly, the second record may include (a) a plurality of qualitydegradation scores for a second compressed version of the particularreference image, obtained using the same PQ algorithms of the third set,and (b) the particular label which was stored in the labeled image dataset for the image tuple of which the second compressed version is amember. Thus, for example, if the third set of PQ algorithms comprisesPQ1, PQ2 and PQ3, and the PQ score generated by a given PQ algorithm PQ1for an image pair (original image I, compressed image C) is representedas Score(PQi, I, C), one record R1 of a training record pair (R1, R2)for a labeled image tuple C1′I1, C2′I1, label L1) may comprise[Score(PQ1, I1, C1′I2), Score(PQ2, I1, C1′I1), Score(PQ3, I1, C1′I1),L1]. The other record R2 may comprise [Score(PQ1, I1, C2′I1), Score(PQ2,I1, C2′I1), Score(PQ3, C2′I1), L1]. In some implementations, because thelabel information is in some sense common to both image pairs of arecord pair, it may not have to be stored separately for each record ofthe pair: for example, a record pair may be provided as input to themachine learning model in a concatenated format similar to [Score(PQ1,I1, C1′I1), Score(PQ2, C1′I1), Score(PQ3, I1, C1′I1), Score(PQ1, I1,C2′I1), Score(PQ2, I1, C2′I1), Score(PQ3, C2′I1), L1].

In at least some embodiments, the record pairs may be used to train amachine learning model to predict, for a post-training input recordcomprising a plurality of quality degradation scores for a particularcompressed version of an image, an output quality degradation score forthe particular compressed version. Note that such a model may not haveto be provided input representing the contents of the images themselves;instead, it may only have to be provided scores generated by other PQalgorithms as input. The model, which may be referred to as a fusionmodel as mentioned earlier because it fuses or combines information frompre-existing PQ algorithms, may comprise a symmetric neural network insome embodiments, in which learned parameters are shared between a pairof sub-models which analyze respective records of a record pair.Although the fusion model technique is described above as using tuplescomprising three images (a reference image and a pair of compressedimages) at several stages, in at least some embodiments individualimages or image pairs may be used instead in one or more of the stagesof the algorithm. Thus, for example, to identify images for whichdifferent PQ algorithms tend to produce divergent PQ scores, such scoresmay be obtained from the different PQ algorithms for one compressedimage at a time (for example, where no reference is available), or forpairs of images (a reference image and a compressed version) at a time.

Pre-existing PQ algorithms may thus be used at several stages of thetraining procedure of the fusion model described above: (a) for theselection of adversarial image sets, (b) for the selection and labelingof non-adversarial image sets, and (c) to produce the scores included inthe training records for the fusion model. In some embodiments, the samecombination of PQ algorithms may be used for each of the three stages;in other embodiments, at least one of the PQ algorithms used for one ofthe stages may not necessarily be used for one or both of the otherstages. Some PQ algorithms may be chosen for one or more stages of thetraining procedure based on their respective resource usagerequirements/metrics: e.g., a PQ algorithm that requires fewercomputation cycles or a smaller amount of memory than other PQalgorithms may be selected in preference to the others, as long as thequality of the fusion model is not affected substantially. In someembodiments, a pre-existing PQ algorithm used for one of the stages mayalso comprise a machine learning model. In some cases, a givenpre-existing PQ algorithm may produce scores at several differentgranularities or at several different intermediate or internal layers:for example, respective intermediate scores generated for differentscales or hierarchical representations of a compressed image may be usedto produce a final PQ score. In at least one embodiment, instead ofusing just the final PQ scores generated by such an algorithm topopulate the training records of the machine learning model, one or moreof the internal or intermediate (non-final) scores may be used topopulate the training records.

In at least some embodiments, as mentioned above, a fusion model of thekind described above may be trained and/or executed at a mediaoptimization service (MOS) of a provider network or cloud computingenvironment. Such an MOS may implement a set of programmatic interfaces,such as a web-based console, command-line tools, application programminginterfaces (APIs), graphical user interfaces and the like, which can beused by MOS clients to submit various types of requests and receivecorresponding responses. For example, an MOS client may use suchprogrammatic interfaces to specify pre-existing PQ algorithms to be usedfor various stages of the training of a fusion model, to specify imagecollections to be used for the training, and so on.

An MOS may support compression and/or other types of processing on mediaobjects such as images, videos, audios, and the like in various fileformats, for presentation of the media objects to clients or end usersin many different contexts. For example, one subset of clients may viewat least some media objects using high-resolution client-side deviceswith substantial computing power available at the client-side devicesfor de-compression and high network bandwidth available for the mediaobjects, while another subset of clients may view at least some mediaobjects using relatively low-compute-power, lower-resolution client-sidedevices and tighter bandwidth constraints. The presentation contexts mayalso differ in the kinds of PQ algorithms which can be used to predictthe perceived degradations of compressed versions of media objects. AnMOS may be used to tailor or tune compression algorithms that can beused for a variety of presentation contexts and/or media formats in atleast some embodiments.

According to some embodiments, a system may comprise one or morecomputing devices, e.g., at an MOS of a provider network. The computingdevices may include instructions that upon execution on or across one ormore processors of the computing devices cause the one or more computingdevices to obtain an indication, at a media optimization service, of oneor more properties of an image presentation context, including (a) animage file format and (b) a type of interface used to present images toviewers. Based at least in part on the properties of the presentationcontext, in at least some embodiments one or more PQ algorithms to beused to tune hyper-parameter values of one or more compressionalgorithms to be used for the context may be selected.

A plurality of optimization iterations may be performed using thecomputing devices in various embodiments. A given optimization iterationmay comprise at the least the following stages in some embodiments. In afirst stage, one or more hyper-parameters of a PQ algorithm may be tunedusing an evolutionary algorithm with a first fitness function and anannotated input data set comprising a first plurality of image pairsrepresented in the image file format of the presentation context.Respective annotations for at least some image pairs of the input dataset may have been generated in a compression quality evaluation teststage of an earlier optimization iteration in some embodiments.

In a second stage of the optimization iteration, one or morehyper-parameters of a compression algorithm selected for the imagepresentation context may also be tuned using an evolutionary algorithmin some embodiments. The fitness function and the input data set usedfor this stage may differ from the fitness function and the input dataset used for the first stage in various embodiments. For the tuning ofthe selected compression algorithm, the input data set may compriserespective perceptual quality scores, generated using a tuned version ofthe PQ algorithm obtained in the first stag, for a second plurality ofimage pairs. A given image pair for which scores are generated using thetuned PQ algorithm may include (a) an image compressed using aparticular hyper-parameter combination of the selected compressionalgorithm and (b) an image compressed using a reference imagecompression algorithm.

According to at least some embodiments, a third stage of theoptimization iteration may comprise conducting a compression qualityevaluation test on a set of images generated using the tuned version ofthe compression algorithm obtained in the second stage of the iteration.The compression algorithm may be evaluated using a new data set whichhas not been used earlier in the optimization iterations, for whichlabels may be obtained from a set of annotators. A result of thecompression quality evaluation test may be used to determine whetheradditional optimization iterations are to be performed; if so, some ofthe labeled data generated in the third stage of the current iterationmay be employed in various embodiments to tune a PQ algorithm in thefirst stage of the next iteration. If no additional iterations arenecessary, e.g., if the compression algorithm has been tunedsufficiently to satisfy a pre-selected quality criterion, the iterativeoptimization procedure may be terminated. Images encoded using a tunedversion of the compression algorithm (obtained, for example, from thefinal optimization iteration) may be presented via the interfaces usedfor the presentation context for which the algorithm was optimized.

In at least some embodiments, a client of the MOS may specify variousconstraints to be used to optimize the compression algorithms—e.g., thekinds of decoders or de-compression algorithms expected to be availableto consumers of the images may be indicated by a client, limits on theamount of CPU or time that can be used for compression or de-compressionmay be indicated, and so on. In such embodiments, the constraints may beused to guide the implementation of one or more of the stages of theiterative optimization procedure—e.g., only certain combinations ofcompression hyper-parameters compatible with the expected decompressorsmay be considered, or the amount of computing resources provided for thecompression algorithm may be limited, and so on.

Any of a number of approaches may be taken with respect to the tradeoffsbetween (a) compressed image size and (b) the perceptual quality of thecompressed images in different embodiments. In one embodiment, forexample, perceptual quality may be maximized while satisfying acompressed image file size constraint. In another embodiment, thecompressed image file size may be minimized while satisfying aperceptual quality threshold constraint. A combination of both theseapproaches may be used in some embodiments. Broadly speaking, thepenalty function used for the optimization tries to minimize/maximizethe objective while increasing/decreasing the penalty if constraints areviolated. Violation of the constraints may be punished harshly (e.g., byrejecting the solution entirely) in some embodiments; in otherembodiments in which a less harsh (or “fuzzy”) approach is used, someviolations of thresholds may be permitted. As mentioned earlier, in someembodiments, evolutionary algorithms may be used in the first and secondstages (for tuning the PQ algorithms and for tuning the compressionalgorithms, respectively) of the optimization iterations;non-evolutionary optimization strategies may be used in at least one ofthe stages in other embodiments. In some embodiments in whichevolutionary algorithms are used in both stages, the algorithms for thefirst and second stages may differ from one another in several ways. Afitness function used for the evolutionary algorithm of the first stagemay be based at least in part on a metric of disagreement between (a)intra-image-pair quality preferences indicated by one or more annotatorsand (b) corresponding intra-image-pair quality preferences generated bythe PQ algorithm. A fitness function used for the evolutionary algorithmof the second stage may be based at least in part on the file sizes aswell as the PQ scores of the compressed images being considered in someembodiments. Different tradeoffs between exploration (tryingthus-far-untried values of hyper-parameters) and exploitation (re-usinghyper-parameter values which have been identified as beneficial) may beused to select parameters of the evolutionary algorithms for the twostages in some embodiments.

In much of the remainder of this document, still images are used as theprimary examples of the kinds of media objects which can benefit fromfusion models and the iterative optimization methodology for compressionalgorithms. It is noted that in various embodiments, analogoustechniques may be used with equal success to enhance predictions of theperceived quality of, and/or tune hyper-parameters of compressionalgorithms for, other types of media objects such as video and audio aswell as (or instead of) still images.

Example System Environment

FIG. 1 illustrates an example system environment in which anetwork-accessible service for optimizing the process of compressingmedia content for over-the-network presentation may be implemented,according to at least some embodiments. As shown, system 100 maycomprise various resources and artifacts of a provider network 101,including a media optimization service (MOS) 110 as well as severalcontent presentation services (CPSs) such as 150 A and 150 B. Resourcesof the MOS 110 may be used to perform numerous types of tasks related tothe optimized presentation of content, including the training andexecution of machine learning models for prediction of perceived qualityof compressed images as well as the tuning of compression algorithms inthe depicted embodiment. A wide variety of compression algorithms 120may be supported by the MOS 110 in the depicted embodiment, andtuned/optimized for various media presentation contexts on behalf ofclients. Numerous perceptual quality algorithms 122 may also besupported by the MOS 110, and executed on behalf of MOS clients and/orto optimize the compression algorithms 120. Other services implementedat the provider network 101, including CPSs 150, may utilize results andrecommendations provided by the MOS 110, e.g., to set the combinationsof hyper-parameter values to be used with a particular compressionalgorithm for various types of media objects that are eventuallypresented to media consumption interface devices (MCIDs) 186.

The term “provider network” (sometimes simply called a “cloud”) refersto a large pool of network-accessible computing resources (such ascompute, storage, and networking resources, applications, and services),which may be virtualized or bare-metal. The cloud can provideconvenient, on-demand network access to a shared pool of configurablecomputing resources that can be programmatically provisioned andreleased in response to customer commands. These resources can bedynamically provisioned and reconfigured to adjust to variable load. Theresources of a provider network may in some cases be distributed acrossmultiple data centers, which in turn may be distributed among numerousgeographical regions (e.g., with each region corresponding to one ormore cities, states or countries). For example, a cloud provider networkcan be formed as a number of regions, where a region is a geographicalarea in which the cloud provider clusters data centers. Each region caninclude two or more availability zones connected to one another via aprivate high speed network, for example a fiber communicationconnection. An availability zone refers to an isolated failure domainincluding one or more data center facilities with separate power,separate networking, and separate cooling from those in anotheravailability zone. Preferably, availability zones within a region arepositioned far enough away from one other that the same natural disastershould not take more than one availability zone offline at the sametime. Customers can connect to availability zones of the cloud providernetwork via a publicly accessible network (e.g., the Internet or acellular communication network). A provider network may include a numberof network-accessible services, such as a virtualized computing service(VCS), one or more storage services, database services, mediaoptimization services such as MOS 110, CPSs 150 and the like. Someprovider network services, such as MOS 110 and/or CPSs 150, may be builtusing other services such as a VCS in the depicted embodiment—e.g.,compute instances of the VCS may be used to implement optimizationtechniques and/or fusion model training of the MOS, or for run-timecontent presentation coordinators such as 156A and 156B of the CPSs. AVCS may also be referred to as an elastic compute service, virtualmachines service, computing cloud service, compute engine, and/or cloudcompute in various implementations.

The MOS 110 may implement a set of programmatic interfaces 177 in theembodiment shown in FIG. 1. Interfaces 177 may include, for example, aset of APIs, one or more web-based consoles, command-line tools,graphical user interfaces and the like. Clients of the MOS 110 maysubmit various types of requests and messages to the MOS from clientdevices 170 (e.g., laptops, desktops, mobile computing devices and thelike), and receive corresponding responses.

An MOS client may use programmatic interfaces 177 to submit a request totrain a machine learning model to generate PQ predictions in someembodiments. In response, a machine learning model which uses amultiple-quality-score-input fusion algorithm 125 may be generated bymodel training and testing coordinators 132 of the MOS 110. In at leastsome embodiments, the machine learning model, which may be referred toas a fusion model, may comprise a symmetric neural network as discussedbelow in further detail in the context of FIG. 7. The process ofpreparing a training data set for the fusion model may comprise data setselection managers 128 identifying, from an image corpus indicated bythe client, a set of image tuples for which selected other PQ algorithms197 generate PQ scores which meet a divergence threshold. Labels forsuch adversarial or divergent image tuples may be obtained from a set ofhuman annotators in at least some embodiments. The MOS 110 may include aset of annotation managers 134 which are responsible forselecting/identifying an initial set of human annotators, e.g., using acrowdsourcing service of the provider network 101 and/or a list of imagequality specialists maintained in knowledge base 138. In addition, in atleast some embodiments, a set of non-adversarial image tuples identifiedby the data set selection managers 128 may be labeled automatically bythe annotation managers 134 using a selected set of the other PQalgorithms 197, and added to the labeled adversarial image tuples toform a labeled image data set. Training records for the fusion model125, comprising respective PQ scores generated using a plurality ofother PQ algorithms 197 may be obtained by the training/testingcoordinators 130. The fusion model may then be trained using thetraining records, and executed as needed by model execution coordinators132 (e.g., in response to client requests for PQ scores for specifiedimage pairs, or as part of iterative optimization of compressionalgorithms 120 ).

In response to compression algorithm optimization requests submitted byMOS clients via interfaces 177, an iterative evolutionary optimizationtechnique may be implemented by optimization coordinators. One or morePQ algorithms 122 (which in some cases may include the fusion algorithm125) may be identified for an image presentation context indicated bythe client. Image presentation contexts may differ from each other inproperties such as the image file format being used, the kinds ofconsumption interface devices 186 expected to be used by the viewers,constraints on the delay that can be tolerated for the display of arequested image, and so on. One or more optimization iterations may beconducted for the compression algorithm, with each iteration comprisingseveral stages. In the first stage, a selected PQ algorithm 122 may betuned (i.e., hyper-parameters of the PQ algorithm may be selected) usinghuman-annotated image sets and an evolutionary algorithm 126 with afirst fitness function in the depicted embodiment. In a second stage,the tuned PQ algorithm may be used to generate scores for imagescompressed using a variety of compression algorithm hyper-parametercombinations, and an evolutionary algorithm 126 may again be used totune the compression algorithm, e.g., with a fitness function that takesthe PQ scores as well as compressed image file size into account. In afinal stage of the optimization iteration, the tuned compressionalgorithm may be evaluated, using a new set of human-annotated imagesets. Statistical techniques may be used to compensate for biases of theannotators in the evaluation stage in at least some embodiments. Ifadditional optimization iterations are required, the PQ algorithm tuningstage of the next iteration may make use of some of the labeled imagesets used in the evaluation stage of the current iteration in at leastsome embodiments.

A given content presentation service (CPS) such as 150A or 150B maypresent requested content, including media objects such as images, fromcontent repositories 152 (e.g., 152A or 152B) to content consumers inthe depicted embodiment. CPS 150A may, for example, present e-book ore-magazine content to its end users, while CPS 150B may provide imagesof products offered for sale by an e-retail organization. Informationabout the manner in which content is being consumed (e.g., propertiessuch as screen size, screen resolution, compute capabilities,de-compression algorithms 187, etc.) of the media consumption interfacedevices (MCIDs) 186 being used) by end users may be stored on contentconsumption metadata 154A or 154B in some embodiments, and such metadatamay be used to characterize the image presentation contexts for whichcompression algorithms are optimized. Recommendations about the tunedversion of the compression algorithms (e.g., the best combination ofhyper-parameter values that have been identified in the iterativeoptimization procedures described above for a given image presentationcontext) may be provided by compression algorithm optimizationcoordinators 136 to content presentation coordinators 156 (e.g., 156A or156B) at the request of MOS clients in the depicted embodiment. Theoptimized settings of hyper-parameters may be used to present images andother content to end users, either directly from the CPS (as in the caseof MCID 186A), or indirectly via edge content servers (ECSs) such as185A or 185B in the depicted embodiment. In the indirect path, thecontent may be transmitted to the ECSs from the CPS, cached in at leastsome cases at the ECSs 185, and transmitted to the end users' MCIDs 186as needed from the ECSs 185. Tuned versions of compression algorithmsmay be used for both steps of such indirect transfers in someembodiments: e.g., a first set of tuned hyper-parameter valuesidentified at the MOS may be used to transfer content from a CPS to anECS, and a different set of tuned hyper-parameter values identified atthe MOS may be used to transfer content from an ECS to end-users' MCIDs.A wide variety of MCIDs 186 may be employed by content consumers indifferent embodiments—e.g., MCID 186K and 186P may comprise a tabletcomputing device, MCIDs 186L and 186A may each comprise a smart phone,MCID 186Q may comprise a laptop or a television, and so on.

Compression algorithm optimization coordinators 136, annotation managers134, model training/testing coordinators 132, model executioncoordinators 132, data set selection managers 128, and contentpresentation coordinators 156 may each be implemented using somecombination of software and hardware of one or more computing devices inthe depicted embodiment. Any of a wide variety of storage devices orstorage services may be used to store data sets used at the MOS, contentrepositories 152, content consumption metadata 154, entries of knowledgebase 138, as well as program code used to implement compressionalgorithms 120, PQ algorithms 122, and evolutionary algorithms 126 invarious embodiments. Additional details regarding various aspects of thefunctionality of MOS 110 are provided below.

Example Factors Influencing Perceived Image Quality

FIG. 2 illustrates example factors which may influence the perceivedquality of a compressed image, according to at least some embodiments.Steps of an example workflow which may be implemented to present animage to an image viewer 295 are shown in the upper part of FIG. 2. Anoriginal (e.g., uncompressed) image 212 may be obtained from an imagerepository 210, and a compressor (also referred to as an encoder) maygenerate a compressed version 216 of the original image using selectedhyper-parameter values of a compression algorithm (e.g., hyper-parametervalues tuned using the iterative approach described above). Thecompressed version 216 may be transmitted via one or more network(s) 215(e.g., the public Internet, and/or a private network) to a mediaconsumption interface device (MCID) 218. The MCID 218 may comprise ade-compressor or decoder 218 compatible with the compressor/encoder 214in various embodiments—that is, a de-compression algorithm that is ableto process the compressed version to produce a reconstructed/decodedversion 220 of the compressed image may be required at the MCID 218. Theimage viewer 295 may form an opinion of the quality 217 of thereconstructed version 220. For example, as shown, the image viewer maythink that the image looks great, okay, poor or terrible. Such opinionsare of course inherently subjective to some extent, and may this differfrom one image viewer to another for the same compressed image, whichmakes the task of estimating perceived quality in an automated way evenmore difficult. Unfortunately, analysis of the physical differences(e.g., expressed as peak signal-to-noise ratio or PSNR) between theoriginal and compressed images is often insufficient to accuratelypredict human-perceived degradation in quality. This is because relyingon such physical differences does not take into account that (a) somephysical differences may be imperceptible to a human in a given viewingenvironment and (b) approximately equal physical differences cansometimes produce very different visual effects/results.

Example factors 250 which can influence the perceived image quality 217may include, among others, the original image quality 251, thecompression algorithm 252 (including the hyper-parameters used), thennetwork transmission quality 253, characteristics 254 of the MCID, theexternal environment 255 in which the image is viewed (such as theambient light conditions, the angle at which the MCID is viewed, etc.),as well as the biological visual system of the image viewer 256 (thehealth of various parts of the eyes, etc.). Different PQ algorithms,which are intended to predict the extent of the perceived degradation(if any) in image quality relative to the original, may try toincorporate or model one or more of the factors 250 in variousembodiments based on various assumptions regarding the relativeimportance of the factors, and/or by weighting physical differences bythe predicted visibility of the physical differences to humans. Forexample, some PQ algorithms attempt to model properties of human visualsystems and how they are affected by various properties of areconstructed compressed image; others may attempt to capture the mostsalient or visible features of images and how these features areaffected by the compression algorithms' hyper-parameters. In general, itmay be difficult for pre-existing PQ algorithms to generalize theirpredictions to take all the different factors potentially impactingimage quality perceptions into account; some existing PQ algorithms maythus work best in specific image presentation contexts.

Variation in Relationships Between Compression Ratios and PerceivedImage Quality

FIG. 3 illustrates example variations in the relationship between imagecompression ratios and perceived quality of the images, according to atleast some embodiments. The compression ratio (CR) of a file, as used inthe context of FIG. 3, is the ratio of the size of an uncompressed orreference version of the file to the size of the compressed version. Asthe CR increases, the size of the compressed version decreases. Graph355A of FIG. 3 shows a measure of human-perceived quality of compressedimages (PQCI) (obtained, for example, using human annotators) along theY-axis, and CR along the X-axis. Generally speaking, from theperspective of a content presentation service (similar to CPSs 150 ofFIG. 1), higher PQCIs are better than lower PQCIs (as they tend toimprove end-user experience), and higher CRs are better than lower CRs(as they tend to reduce network bandwidth requirements and latencies).

Graph 355A shows the PQCI-versus-CR curve averaged over all the imagesof an example data set. At low CRs, images may be perceived to be ofhigh quality, and at very high CRs, the images may be perceived to be oflow quality, but the extent of perceived quality degradation may not belinearly related to the extent of compression. Furthermore, if oneclassifies the images of a given data set into categories (such ascategory 1 for faces, category-2 for daylight landscapes, category-3 forfurniture items, etc.), and plots the PQCI-versus-CR curves for eachcategory separately, the shapes of the category-specific curves maydiffer from the shape of the overall curve of FIG. 355A. In Graph 355B,for example, representing category-1, the falloff in PQCI 310A may besteeper than in graph 355A for smaller CRs. In graph 355B forcategory-2, the falloff in PQCI 310B is generally smaller than thatshown in graph 355A, while for category-3 images, graph 355D shows thatthe maximum reduction in PQCI may be less than that shown in graph 355A.Note that the graphs shown in FIG. 3 are synthetic examples presented toshow that such variations in relationships between compression ratiosand perceived image quality may exist, and may not necessarilycorrespond to actual measurements for any given data set or categories.The fact that the extent to which compression affects perceived qualitymay vary for different categories of images may be used at a mediaoptimization service similar to MOS 110 of FIG. 1 in at least someembodiments to recommend custom image-category-specific compressionalgorithm hyper-parameter combinations as discussed below in furtherdetail.

Overview of PQ Algorithms

As indicated earlier, a suite of existing PQ algorithms may be employedfor training machine learning models that combine the technicaladvantages of existing PQ algorithms in various embodiments. Beforeconsidering the details of the techniques for training such models, itmay be helpful to provide general information about the way in which PQalgorithms may be used. FIG. 4 illustrates example inputs and outputs ofa perceived quality degradation estimation algorithm which may be usedto select compressed images for presentation to end users, according toat least some embodiments.

In the depicted embodiment, a perceived quality degradation estimationalgorithm 414 may consume two pairs of images, 410A and 410B, as input,and produce respective PQ scores 416A and 416B for each pair. Pair 410Amay comprise an original image x0, and a compressed version x1 of x0,obtained for example from some compression algorithm C1 using a set ofhyper-parameter values h1. Pair 410B may comprise the same originalimage x0, and a different compressed version x2 of x0. x2 may beobtained from a different compression algorithm and hyper-parametercombination than x1—e.g., using the same compression algorithm C1 anddifferent hyper-parameters h2, or a different compression algorithm C2and hyper-parameter values c3.

Algorithm 414 may generate PQ score 416A indicating the estimated orpredicted extent of the degradation of x1 relative to x0, and PQ score416B indicating the estimated or predicted extent of the degradation ofx2 relative to x0. Based on the generated scores, a preferred compressedversion 418 of x0 may be selected, and provided by an image presentationsubsystem 425 to one or more viewers. The selection of the preferredimage 418 may be based on a tradeoff between the PQ scores and the imagesizes of the compressed versions. For example, in one embodiment, inorder to choose a preferred image, a first weight W1 may be assigned tothe difference in PQ scores 416A versus 416B, and a second weight W2 maybe assigned to the difference in compressed file sizes, and a finalpreference score may be computed as the sum of the weighted values. Suchapproaches which combine file size and PQ scores may also be used in theiterative evolutionary optimization of compression algorithms asdiscussed below in further detail. Note that in some cases, a PQalgorithm 414 may of course be used to estimate the extent ofdegradation of perceived quality for a single compressed version of animage, so the input to the PQ algorithm may not necessarily comprise twocompressed versions of the same image.

Example Algorithm-to-Algorithm Variations in PQ Scores

As suggested earlier, a variety of PQ algorithms have been developedover the years, with different underlying assumptions about how best tomodel factors influencing perceived quality. In some cases, the PQscores produced by respective algorithms for the same input images mayvary substantially. FIG. 5 illustrates example variations in qualityscores produced by different perceptual quality algorithms, according toat least some embodiments.

Respective PQ score sets 516A, 516B and 516C may be generated for thesame image tuple 510 in the depicted embodiment by PQ algorithms 514A,514B and 514C. Tuple 510 may comprise a reference image x0 and a pair ofcompressed versions x1 and x2 of the image. Score set 516A may predictthat x1 would be considered far superior to x2 by viewers. Score set516B may predict that viewers would consider x1 and x2 to be of aboutthe same quality, while score set 516C may predict that x2 would beconsidered superior to x1.

Graph 555 shows example distributions of agreement scores 590 of the PQalgorithms with respect to the opinions of human annotators on thequality of a collection of image tuples. Note that these are simulateddistributions, shown here to indicate the kinds of divergence exhibitedby different PQ algorithms, and do not correspond to results obtainedwith any given data set. An agreement score for a given image tuple anda given PQ algorithm is a measure of the similarity of the PQ scoregenerated by that PQ algorithm to the annotations/opinions expressed onaverage by the human annotators for that same image tuple. An agreementscore for a given image tuple and the human annotators 518 is a measureof the similarity of the annotations/opinions expressed by the differentannotators for that image tuple. The Y-axis of graph 555 representsagreement scores, plotted on a scale of 0.0 to 1.0, with 1.0 indicatingperfect agreement and 0.0 indicating complete disagreement. Four rangesof agreement scores are shown, for human annotators 518 and for each ofthe three PQ algorithms 514A, 514B and 514C. The dark vertical blacklines for each of the four cases indicate the minimum-to-maximum rangeof agreement scores over the collection of image tuples considered, withthe small horizontal marks along the dark black lines indicating theaverage agreement scores computed for each case for the collection ofimage tuples. The range for humans (with the average close to 0.75)indicates the extent of agreement among the opinions of the differenthuman annotators; if all the human annotators had the same opinionsabout all the figures, the range and average agreement score would bothbe represented by a single dot with a Y-axis value of 1.0.

As indicated in graph 555, the ranges and averages of the agreementscores of the three different PQ algorithms may diverge from one anotherfor at least some image tuple sets. As suggested above, such divergenceor disagreement may arise due to the underlying assumptions of the PQalgorithms. One of the PQ algorithms such as 514A may, for example, usean approach based on the MS-SSIM (Multiscale Structural Similarity)technique. MS-SSIM decomposes an image into a multi-scale pyramid(iteratively down-sampling the image). At each scale, the similaritybetween two images is measured in terms of luminance (mean), contrast(variance), and “structure” (covariance) over local regions, which arethen weighted and spatially averaged to produce a final MS-SIM score.

Another of the PQ algorithms such as 514B may be based on the HDR-VDP2(High Dynamic Range Visible Difference Predictor 2) methodology. Thistechnique weights physical image differences according to assumptionsabout aspects of the human visual system, such as intra-ocular lightscatter, opsin dynamics (photochemical processing in the retina),contrast sensitivity across the visible luminance range and contrastmasking. A third PQ algorithm, such as 514C, may be based on the HaarPSItechnique which performs image similarity analysis by measuring phasecoherence in spatial filters. In this approach, phase congruency of Haarwavelets between the compared images in the vertical and horizontaldirections is computed as a measure of similarity over space, and thesimilarity maps thus produced are weighted according to filter amplitudein the luminance channel. The dense weighted similarity map is thenaveraged to produce a final score for the compressed image. A variety ofPQ algorithms, including but not necessarily limited to the algorithmsdiscussed in the context of FIG. 5, may be employed to train a fusionmachine learning model in at least some embodiments and/or to tunecompression algorithms iteratively. Such PQ algorithms may include,among others, SSIM (Structural Similarity), FSIM (Feature Similarity),PSNR (Peak Signal to Noise Ratio), PSNR-HVS (Peak Signal to NoiseRatio-Human Visual System), VDP (Visible Difference Predictor), HDR-VDP(High Dynamic Range Visible Difference Predictor), Normalized LaplacianPyramid, LPIPS (Learned Perceptual Image Patch Similarity), PieAPP(Perceptual Image-Error Assessment through Pairwise Preference) and VSI(Visual Saliency-Induced Index) algorithms.

Overview of Example Technique for Preparing Training Data Set

In at least some embodiments, a specially-curated set of images may beselected to help in the training of a machine learning model forgenerating PQ scores. FIG. 6 illustrates an example technique forpreparing a training data set for a machine learning model forpredicting perceived image quality, according to at least someembodiments. Two types of image tuples may be selected in the depictedembodiment for training such a model. An “adversarial” image tuple setIT1 as well as a “non-adversarial” image tuple set IT2 may be selectedfrom a larger available group of image tuples, as indicated in elements604 and 606 respectively. Each image tuple may comprise at least anoriginal image, a first compressed version of the original, and a secondcompressed version of the original. An image tuple may be designated as“adversarial” if PQ scores generated for the image tuples (e.g., similarto the [x0−x1] and [x0−x2] scores discussed in the context of FIG. 4) byrespective PQ algorithms of a set S1 differ by more than a threshold.Note that the divergence being considered for selecting image tuples forIT1 is the difference in PQ scores generated for the same image pair bythe respective PQ algorithms, not the difference in the PQ scoresgenerated for different image pairs of a given image tuple by any givenPQ algorithm. Essentially, the adversarial image tuples represent casesin which a selected set S1 of PQ algorithms is unable to reach consensus(or near-consensus) regarding which compressed image is likely to becloser in perceived quality to the original.

In contrast, the non-adversarial image tuples of IT2 may be selected forreasons other than divergence in the PQ scores of different PQalgorithms in various embodiments. In fact, in some embodiments, imagetuples for which a set of PQ algorithms S2 (which may be the samealgorithms as those of set S1, or may differ in at least one algorithmfrom S1) agree or provide similar results may be chosen as members ofIT2. The non-adversarial image tuples may be selected so as to capturecases where the choices regarding which of the compressed versions of atuple is more similar to the original are more clear cut.

A set of human annotators 610 may be used to obtain labels for theadversarial image tuples, indicating which of the compressed versionswere found more similar to the original image. The labels may beexpressed by the human annotator 610 s in any of a variety of ways indifferent embodiments: e.g., as a binary value (with a “1” for theless-degraded compressed version and a “0” for the more-degradedcompressed version), or as a numerical value on a scale (e.g., a 10 on ascale from 1-10 for a compressed image which appears indistinguishablefrom the original, and a 1 for a compressed image which appearsextremely dissimilar from the original). In at least some embodiments,the labels generated by different human annotators for a given imagetuple may not be identical; in some such embodiments, a statisticalaggregate (e.g., a mean or median label) may be computed from thedifferent per-annotator labels. In at least one embodiment, analysis ofthe divergence in opinions among the human annotators may be performed,e.g., by annotation managers similar to those shown in FIG. 1). In someembodiments, the labels generated by those human annotators which arefound to disagree extensively from other human annotators may bediscarded or pruned; as such, only consensus or near-consensusannotators' labels may be retained for training the machine learningmodel in such embodiments. In at least one embodiment, the differentlabels generated for any given image tuple by the different humanannotators may be examined for divergence, and outliers among the labelsmay be discarded or pruned on a per-image-tuple or per-image-pair basis.

Labels for the non-adversarial image tuples of IT2 may be generatedautomatically, without using human annotators in at least someembodiments, as indicated in element 612. Such labels may be based onthe PQ scores generated by the algorithms of set S2 in variousembodiments.

A combined labeled image tuple set 620 may be stored in variousembodiments, comprising the human-annotator-labeled image set LIT1derived from IT1, and the automatically-labeled image set LIT2 derivedfrom IT2. Using the images included in the combined image tuple set, anda set of PQ algorithms (which may be the same algorithms as thoseincluded in S1 and/or S2, or may include at least one PQ algorithm whichis not in S1 or S2), training records 625 for the machine learning modelmay be generated in the depicted embodiment. The training records may beorganized as pairs in some embodiments, with each training record of apair comprising respective PQ scores generated for a given (originalimage, compressed version) pair of an image tuple, and a correspondinglabel. The training records 625 may then be used to train a fusion model619, which in effect learns indirectly (through the scores included inthe training records) about subtle quality differences from the labeledimage tuples of LIT1, and about more basic quality contributing factorsfrom the labeled image tuples of LIT2. In at least some embodiments, aneural network based model may be used as the fusion model, as indicatedin FIG. 7.

Overview of Example Architecture of Neural-Network Based Fusion PQ Model

FIG. 7 illustrates an example architecture of a neural network-basedmodel for predicting perceived image quality based on scores generatedby multiple other perceptual quality algorithms, according to at leastsome embodiments. As discussed in the context of FIG. 6, a combinedlabeled image tuple set 720 may be obtained in various embodiments,comprising adversarial as well as non-adversarial image tuples.Individual ones of the tuples may comprise an original image x0, and twocompressed versions x1 and x2 of x0. A set of PQ algorithms 725 (e.g.,725A, 725B, 725C, etc.) may be selected to generate training records forthe neural-network based model. The PQ algorithms may be selected basedat least in part on information about the image presentation context(s)for which enhanced PQ scores are required in some embodiments; suchinformation may be provided programmatically, for example, by clients ofa media optimization service (MOS) similar to MOS 110 of FIG. 1 at whichthe fusion model is to be trained.

A set of training record pairs {Ri, Rj, label} 728 may be generatedusing the selected PQ algorithms and the labeled image tuple set 720 inthe depicted embodiment. Record Ri may contain PQ scores for (x0-x1)generated by each of the PQ algorithms for a given tuple of set 720,while Rj may contain PQ scores for (x0-x2) for the same tuple and thesame PQ algorithms. Ri and the label may be provided as input to onesymmetric sub-model 732A of a neural network based model 730 in thedepicted embodiment, while Rj and the label may be provided as input toa second symmetric sub-model 732B. Learned parameters (such as weights)may be shared among between the sub-models. A learned PQ score scorelmay be generated for the (x0-x1) combination at the sub-model 732A, anda learned PQ score score2 may be generated for the (x0-x2) combinationat the sub-model 732B. Each of the sub-models may comprise some numberN1 of fully-connected layers followed by one or more Softplus layers insome implementations. A fully connected layer may, for example, performa linear transformation y=xA^(T)+b on the input data where A is theweight matrix and b is the bias vector. The output of the linear layermay be passed through a Softplus layer (Softplus(x)=log (1+exp(x)).Softplus allows gradient calculations for negative inputs. Output fromthis Softplus layer may be passed through one or more furthercombinations of linear and Softplus layers in some implementations. Suchrelatively simple neural network architectures may be sufficient in someembodiments because the training data has been curated to provideexamples which capture salient information about images in a givenpresentation context very effectively; if examples which are lessrepresentative of the presentation context were used, or of a singlemodel 730 were expected to provide PQ scores for a wide variety of imagepresentation contexts, more complex neural networks may be required.

In at least some embodiments, within a given image tuple used forgenerating the training records, one of the compressed image versionsmay be replaced by the original or reference version; for such imagetuples, this “fake” compressed image would be expected to be consideredsuperior to a visibly compressed version. A zero-difference enforcementlayer 745 may be included in the model 730 to anchor the scale of thelearned output, which forces the model's output learned score to a valueindicating that the fake compressed image is of a higher quality thanthe compressed image of the same tuple. Thus, for example, if a learnedscore value of zero indicates that the compressed image of a given imagepair is indistinguishable from the reference version, the output learnedscore would be set to zero for the image pair in which the “compressed”version is replaced by the original version. The model 730 may alsoinclude a deterministic preferred image selection probabilitydetermination layer 745 in some embodiments. For example, in oneimplementation, an inverse logit function may be used to determine theprobability that x1 is preferred over x2 (that is, that x1's perceivedquality is closer to that of the original than that of X2). Theprobability that x1 is preferred may be expressed aslogit⁻¹(score2−score1) in such an implementation. Note that after themodel 730 is trained, either sub-model may be used to predict PQ scoresfor a given image pair, and both sub-models may not have to be used.Tunable hyper-parameters of the fusion model 730 may include, amongothers, the number of PQ models used to generate the training records,the specific combination of PQ models selected, how many “fake”compressed images are included in the image tuples, the number oftraining epochs, the training batch size, the learning rate, and thenumber and types of layers of the neural network in the depictedembodiment. Such hyper-parameters may be tuned in some embodiments usingiterative evolutionary techniques as discussed below in further detail.

Methods for Fusion Model Training and Execution

FIG. 8 is a flow diagram illustrating aspects of operations which may beperformed to predict perceived image quality using a fusion model whoseinput comprises respective scores from multiple other perceptual qualityalgorithms on a combination of adversarial and non-adversarial trainingexamples, according to at least some embodiments. As shown in element801, a representative image collection RIC for a particular imagepresentation context or use case may be obtained, e.g., at a mediaoptimization service similar in features and functionality to MOS 110 ofFIG. 1. In one example scenario, for example, a collection of websitelogs (containing entries indicating which images and web pages have beenserved over some time period) of an application may be provided to andanalyzed by an MOS component to determine the types of images that aretypically presented to the application's users. Machine learningalgorithms such as clustering algorithms and/or image recognitionalgorithms may be used to classify the served images, detect the kindsof objects included in the different classes, and so on. Uncompressed orbaseline versions of at least some of the representative images may alsobe obtained in at least some embodiments.

An adversarial image set IT1 may be identified or selected from RIC(element 804) in various embodiments. Image tuples may be selected forinclusion in IT1 if they satisfy a PQ-algorithm-to-PQ-algorithm scoredivergence threshold. That is, image tuples (each comprising a referenceimage and at least two different compressed versions) may be included inIT1 if a set of PQ algorithms tend to disagree regarding which of thecompressed versions is more similar to (and hence less degraded relativeto) the reference image. A set of labels may be obtained for IT1 imagesfrom a set of human annotators in the depicted embodiments (element807), indicating which of the compressed versions of a given tupleappears to be more similar to the reference version. In one embodiment,a subset of the human-generated annotations may optionally be pruned orremoved from further consideration, e.g., due to a lack of consensusamong the different annotators being used. In another embodiment,individual ones of the annotators may be identified as inconsistent ordivergent with respect to other annotators, and such anomalousannotators may be removed from the pool of annotators (and theannotators generated by such anomalous annotators may be discarded).

A second, non-adversarial image tuple set IT2 may be identified invarious embodiments from the RIC, and labeled automatically using PQscores obtained from a set of PQ algorithms (element 810). In at leastsome embodiments, image tuples may be selected for inclusion in IT2based at least in part on consensus or clear-cut preferences/scoresgenerated by the PQ algorithms. A combined labeled image data set LIDS,comprising IT1 and its human-generated labels, as well as IT2 and itsautomatically-generated labels, may be stored in the depicted embodiment(element 813).

Using multiple PQ algorithms and the LIDS, labeled training recordspairs may be generated for a machine learning model ML1, such as aneural network-based model (element 816). Each record of a record pairmay comprise PQ scores from each of the multiple PQ algorithms for arespective compressed version of an image tuple relative to thereference image of the tuple. Some of the labeled records may bewithheld for testing/validating the model in at least some embodiments.ML1 may then be trained using labeled record pairs (element 819) togenerate, for a given post-training input record comprising the PQscores (from the same PQ algorithms used in operations corresponding toelement 816) for a compressed image relative to its reference image, anoutput PQ score indicating the perceived quality degradation of thecompressed version relative to the reference image. The training may beterminated when the model meets prediction accuracy targets selected forthe image presentation context or use case (as indicated using a testingdata set), or when the resources or time used for the training reaches athreshold in some embodiments. The trained version of ML1 may be used toorchestrate the efficient (in terms of file size) presentation of imageswith PQ scores in target desired ranges in the image presentationcontext from which the RIC was obtained (element 822). For example, fromamong several different compressed versions of an image, the image whichhas the smallest file size among those images which satisfy a target PQscore requirement may be selected.

In the procedure described above, adversarial and non-adversariallabeled images are combined and used together to train the fusion model.That is, in a given batch of training records, PQ scores obtained frommultiple algorithms for some labeled adversarial image tuples and somelabeled non-adversarial image tuples may be included. FIG. 9 is a flowdiagram illustrating aspects of operations that may be performed in atwo-stage process to train a machine learning model for perceived imagequality prediction, in which adversarial training examples are used inone of the stages and non-adversarial training examples are used in theother stage, according to at least some embodiments. Operationscorresponding to elements 901, 904, 907 and 910 of FIG. 9 in variousembodiments may be similar to operations corresponding to elements 801,804, 807 and 810 of FIG. 8 respectively: a representative imagecollection may be identified for a context, adversarial image tuple setIT1 and non-adversarial image tuple set IT2 may be identified, IT1 maybe labeled with the help of human annotators and IT2 may be labeledautomatically requiring the help of human annotators.

In operations corresponding to element 913 of FIG. 9, a decision may bemade as to whether the adversarial training data alone is to be usedfirst to train a first version of a fusion machine learning model, whichis then trained further using the non-adversarial training data, orwhether the opposite approach is to be used (training using thenon-adversarial data first, followed by training using the adversarialdata). This decision may be made based on various factors in differentembodiments, such as the number of adversarial and non-adversarial imagetuples which are available, properties of the image presentationcontext, client-specified preferences, and so on. In at least someembodiments, a knowledge base of a media optimization service maycontain entries indicating which training approach (the approach ofcombining adversarial and non-adversarial data as indicated in FIG. 8,adversarial-first training, or non-adversarial-first training) has beenfound most effective for different scenarios or use cases, and suchknowledge base entries may be used to decide how the training should beconducted.

If a decision is made to train a first version of the fusion model usingadversarial training data alone (as indicated in element 916), versionV1 of the model may be trained using training data record pairscorresponding to IT1 images (element 919). It may turn out to be thecase that this first version V1 (also referred to as a pre-trainedversion) of the model satisfies the target prediction accuracy goals forthe use case being considered, in which case no further training may berequired. If, however, the goals have not been met, a second version V2of the model may be trained using record pairs corresponding to labeledIT2 images (element 922) in the depicted embodiment.

Similarly, if a decision is made to train a first version of the fusionmodel using non-adversarial training data alone (as also determined inoperations corresponding to element 916), version V1 (the pre-trainedversion) of the model may be trained using training data record pairscorresponding to IT2 images (element 925) in at least some embodiments.If this first version V1 of the model satisfies the target predictionaccuracy goals for the use case being considered, no further trainingmay be required. If, however, the goals have not been met, a secondversion V2 of the model may be trained using record pairs correspondingto labeled IT1 images (element 928) in the depicted embodiment. Thefinal trained version of the model may be stored in various embodiments,and used to orchestrate the efficient presentation of image with PQscores in desired ranges in the image presentation context for which themodel was trained (element 931).

FIG. 10 illustrates example improvements in results which may beachieved using a fusion model for predicting perceived image quality,according to at least some embodiments. Graph 1055 of FIG. 10 shows thesame information as was presented in graph 555 of FIG. 5, but withadditional agreement scores plotted for a fusion model trained usingmethodology similar to that depicted in FIG. 8. For the pairs oftraining records used to train the fusion model (whose results areindicated by label 1018), scores generated by PQ algorithms 514A, 514Band 514C for a labeled image tuple set were used.

As shown, the fusion model's agreement score results with respect tohuman annotators are superior, at least on average, to those of each ofthe PQ algorithms which were used to train the fusion model in thedepicted embodiment. The extent of the improvement in agreement scoresobtained using the fusion approach may of course vary with the PQalgorithms and the image presentation context being considered invarious embodiments. In some cases, for a given image presentationcontext and corresponding representative image set, the best agreementscores achieved by a given PQ algorithm used for generating the trainingdata may be superior to some of the agreement scores achieved using thefusion methodology—e.g., the low end of the fusion model agreement scorerange 1018 overlaps with the high ends of the ranges for PQ algorithms514A and 514B.

Example Programmatic Interactions Associated with Fusion Models

As mentioned earlier, in various embodiments a media optimizationservice may implement programmatic interfaces to enable clients tosubmit requests pertaining to compression techniques and receivecorresponding responses. FIG. 11 illustrates example programmaticinteractions associated with the training of a fusion model forpredicting perceived image quality, according to at least someembodiments. Programmatic interfaces 1177 implemented by MOS 1112(similar in functionality to MOS 110 of FIG. 1) may include a set ofAPIs, web-based consoles, command-line tools, graphical user interfacesand the like in different embodiments.

A client 1110 may submit an IdentifyRICFromUseCaseLogs message 1114 tothe MOS 1112, providing a pointer to some collection of logs whichindicate the kinds of images that were served to end users of anapplication or service over some period of time, from which the MOS isbeing requested to select representative image collections (RICs) forone or more image presentation contexts represented in the logs in thedepicted embodiment. The MOS 1112 may extract the list of images fromthe log files, along with information about the temporal distributionsof the presentations of the images, and analyze the extractedinformation to select a subset of images as representative of thecontexts or use cases. A RepresentativelmagesSelected response message1115 may be sent to the client in some embodiments after therepresentative images have been identified.

In some embodiments, a client 1110 may submit anIdentifyAdversarialImageTuples request 1117 to the MOS, requesting theselection of image tuples from a specified RIC for which a selected setof PQ algorithms tend to disagree on their predicted PQ scores. In someembodiments, the client 1110 may indicate the particular PQ algorithmswhose scores for the same image tuples are to be compared to identifythe adversarial tuples. In other embodiments, the MOS may select the PQalgorithms, e.g., using a knowledge base. The selected orclient-specified PQ algorithms may be used to generate respective PQscores for at least a portion of the RIC in the depicted embodiment. Thescores produced by the different algorithms may be examined to identifythose tuples whose inter-algorithm score divergence exceeds a threshold,and such tuples may be included an adversarial image tuple set. Thethreshold itself may be indicated by the client 1110 in someembodiments, or selected by the MOS 1112. In some embodiments, adivergence criterion rather than a precise threshold may be used—e.g.,the image tuples whose divergence measures are in the top N % among thedivergence scores being examined may be included in the adversarialimage tuple set. An AdversariallmageTuplesIdentified response message1121 may be sent to the client to the client after the adversarial imagetuple set has been selected in at least some embodiments.

A client 1110 may submit an ObtainAnnotationsForAdversarialTuplesrequest 1123 to the MOS in some embodiments, indicating thathuman-provided annotations are to be obtained for a specified set ofadversarial image tuples. In response, annotation manager components ofthe MOS 1112 (similar to annotation managers 134 of FIG. 1) mayorchestrate an annotation procedure. Such a procedure may, for example,comprise identifying/selecting available annotators, providing labelinginstructions to the annotators, setting up the environments in which theannotators are to view the image tuples or image pairs, presenting theimages of the specified set to the annotators in those environments,obtaining the annotations from the annotators, and post-processing theresults in at least some embodiments. In some cases, for example, theinter-annotator consistency among the annotators may be analyzed todetermine a measure of the extent of agreement among different humanannotators, and outliers of one or more kinds may be identified in thepost-processing phase. One type of outlier may comprise image tuples forwhich different human annotators tend to disagree; in some embodiments,at least some such outlier image tuples may be removed or excluded fromthe labeled adversarial image tuple set. Individual annotators whoseopinions tend to diverge consistently from those of other annotators mayrepresent another class of outlier identified by the MOS 1112 in someembodiments; some or all of the labels produced by such outlierannotators may be discarded when preparing the labeled adversarial imageset in such embodiments. After the annotations have been obtained andpost-processed, an AnnotationsObtained message 1125 may be sent to theclient 1110 in some embodiments.

In at least one embodiment, a client may submit anIdentifyNonAdversarialImageTuples request 1128 to the MOS viaprogrammatic interfaces 1177. Such a request may indicate representativeimage collection from which a non-adversarial image tuple set (i.e., acollection of image tuples for which different PQ algorithms tend toagree) is to be identified. In some embodiments in which scoresgenerated by different PQ algorithms for representative images havealready been obtained from the same representative image collection inthe process of selecting adversarial images, some or all of those samescores (the ones representing strong inter-PQ-algorithm agreement ratherthan disagreement) may be used to select the members of thenon-adversarial image tuple set. The client may in some cases useparameters of request 1128 to indicate which PQ algorithms are to beused to select non-adversarial tuples. If scores from the PQ algorithmsto be used for the selection of non-adversarial image tuples have notalready been obtained, the specified PQ algorithms may be run withrespect to the specified representative images and the tuples for whichthe agreement among the PQ algorithms exceeds a threshold may beselected in the depicted embodiment. After the non-adversarial imagetuples have been selected, a NonAdversariallmageTuplesldentified message1133 may be sent to the client 1110 in some embodiments. In at least oneembodiment, a client 1110 may provide a set of non-adversarial imagetuples to the MOS, and/or a set of adversarial image tuples, instead ofrequesting the MOS to select the sets.

A client 1110 may submit an AutoAnnotateNonAdversarialTuples request1141 in some embodiments, requesting that annotations be generatedautomatically (e.g., using scores obtained from one or more PQalgorithms) at the MOS for a specified set of non-adversarial imagetuples. Such annotations or labels may be obtained at the MOS, e.g., byan annotation manager without the help of human annotators, and anAutoAnnotationsGenerated message 1143 may be sent to the client.

In at least some embodiments, a client 1110 may submit aTrainFusionModel request 1145 to the MOS 1112, requesting that a machinelearning model (e.g., similar to the neural network-based model of FIG.7) be trained on behalf of the client. In some embodiments, the request1145 may indicate the specific PQ algorithms to be used to generate thescores included in the training records for the model, and/or values forother hyper-parameters values such as the initial learning rate, thebatch size, the number of epochs, etc. In one embodiment, a client maysimply provide an indication of a representative image collection to beused to train a fusion model in a TrainFusionModel request, and the MOS1112 may perform the various needed tasks automatically, includingidentifying and obtaining annotations for an adversarial andnon-adversarial set of image tuples and then training the model. In suchan embodiment, separate requests 1117, 1123, 1128 and/or 1141 may not berequired. Some clients may prefer to let the MOS decide details, such aswhich PQ algorithms are to be used at each stage, while others mayprefer to indicate their preferences regarding one or more of the stepsinvolved in training the model. After the fusion model has been trained,the trained version may be stored, and a ModelTrained message 1147 maybe sent to the client in some embodiments.

A client 1110 may request that PQ scores be predicted using the trainedversion of the fusion model on a specified collection of one or imagepairs or tuples by submitting a PredictPQScores request 1151 in someembodiments. The model may be executed by the MOS 1112, and thepredicted scores generated may be provided to the client in one or morePredictedScores messages 1153. In at least some embodiments, resultsobtained from the trained model may be used to select hyper-parametersof a compression algorithm used to present images from a web site orother content presentation environment. The output of the trainedversion of the model in such a scenario may comprise (a) a first set ofquality degradation scores for a first set of compressed images producedusing a first set of hyper-parameters of a compression algorithm, and(b) a second set of quality degradation scores for a second set ofcompressed images produced using a second set of hyper-parameters of thecompression algorithm. The output quality degradation scores for the twohyper-parameter value combinations may be compared, and the combinationwhich led to superior perceived image quality may be used to present aset of images to viewers. In some embodiments, fusion model-relatedprogrammatic interactions other than those shown in FIG. 11 may besupported by an MOS 1112.

Example Image File Formats and Compression Algorithm Sub-Operations

A media optimization service may support the analysis and tuning of awide variety of compression algorithms, associated with numerous fileformats in some embodiments. The compression algorithms may comprise apipeline of several stages or sub-operations in some cases, several ofwhich may have associated hyper-parameters which can be tuned fordifferent image presentation contexts at the optimization service inresponse to client requests. FIG. 12 illustrates example compressedimage file formats and compression algorithm sub-operations, accordingto at least some embodiments. File formats 1210 for which compressionalgorithm optimization is supported at an MOS may include, among others,JPEG 1211, WebP 1212 (a format derived from the VP8 video file format),AVIF (a still image file format based on AOMedia Video 1) 1213, and MP4(Moving Pictures Expert Group—4) 1224 in the depicted embodiment.

A compression algorithm pipeline may comprise operations or stages ofseveral different types 1220 in different embodiments. Such stages mayinclude, depending on the specific compression algorithm and fileformat, chroma subsampling 1221, block prediction 1222, transformationto frequency domain 1223, quantization 1224, run-length encoding 1225,and the like.

In some image compression techniques, the image may be converted to theYUV color space, which has one brightness channel (luma) and two colorchannels (chrominance). Chroma sub-sampling (such as YUV420, in whichthe U and V channels have half the resolution of the Y channel) may beapplied to the chrominance channels, effectively reducing the resolutionof the chrominance images and taking advantage of the fact that thehuman visual system is less sensitive to fine changes in color relativeto brightness. JPEG offers chroma subsampling as an option, while WebPenforces it. Settings for whether chroma sub-sampling is to beperformed, and if so, the specific resolution ratios for the chrominancechannels relative to the luma, may constitute one set ofhyper-parameters for a compression algorithm.

With advances in video codec technology, many ideas for videocompression have been adapted to still images. Intra-frame blockprediction became popular through WebP, which is derived from the V8video codec. In block prediction, a respective prediction model isapplied to sub-blocks of a macro block, effectively performingfiltering. The best predictor for a given block is identified using themodel, and the output (residuals) are then quantized.

In some image compression techniques, the image obtained after colorspace transformation is divided into pixel blocks (e.g., 8×8 pixelblocks with JPEG and 16×16 or 8×8 pixel blocks in WebP) which are thentransformed into the frequency domain. For example, a discrete cosinetransforms (DCT) algorithm is used for JPEG and WebP and a discretewavelet transforms algorithm is used for JPEG2000. In some cases, aswith AVIF, any of multiple transform algorithms may be used. Conversioninto the YUV space and a loss of precision in the transformation to thefrequency domain can lead to distortions. The image blocks after theconversion may be quantized, e.g., according to respective quantizationtables for the different channels. For JPEGs, the DCT terms may bedivided by the corresponding value in a quantization table, and roundedto the nearest integer. Higher values in the quantization tables maylead to higher probabilities that the results of the division step arezero. For WebP, the input for quantization comprises block predictions,not the raw image data itself. In some implementations, afterquantization tables have been tuned (e.g., using the iterativeevolutionary approach discussed below in further detail), the tunedtables may be transmitted for use at the media consumption interfacedevices.

JPEG compression uses run-length zig-zag encoding (RLE) of the 8×8frequency coefficient blocks, taking advantage of sparsity patterns inthe coefficient blocks. RLE transforms a sequence of values intosequences of pairs. The first element of a pair is called a symbol, andthe second element is a non-zero value. For each sequence of values, thesymbol is used to code the number of preceding zeros and the bit lengthof the non-zero value. In RLE, one value is used to indicate the numberof consecutive zeros before the next non-zero value in the data. JPEGimages can be further optimized through progressive encoding, and maybenefit from Huffman or arithmetic coding as well. WebP utilizesarithmetic coding.

When considering the choices for a compression pipeline, contentpresentation services may have to select values for a number ofhyper-parameters (e.g., on the order of dozens or hundreds of values forquantization tables alone) for the various stages of the pipeline atwhich some combination or all of the operation types shown in FIG. 12may be performed. In various embodiments, the factors to be taken intoconsideration may include encoding (compression) and decoding speeds,the compression ratios achieved, as well as compatibility of theexpected media consumption interface devices with the compressionalgorithms—for example, not all browsers at some smart phones or tabletdevices may be able to perform complex computations needed toreconstruct images compressed using some compression pipeline settings.

Overview of Iterative Compression Algorithm Optimization

FIG. 13 is a flow diagram illustrating a high-level overview of aniterative compression algorithm optimization technique, according to atleast some embodiments. The approach presented in FIG. 13 resolves the“chicken-and-egg” problem of tuning compression algorithms mentionedearlier: namely, that in order to tune a compression algorithm in anautomated manner, one usually needs a sufficiently-tuned PQ algorithm,but the process of tuning the PQ algorithm may itself require using asufficiently-tuned compression algorithm.

A compression algorithm whose hyper-parameters are to be tuned using oneor more PQ algorithms for a given image presentation context may beidentified. In operations corresponding to element 1301, which representa first phase of a three-phase optimization iteration, the PQ algorithmsmay themselves be tuned without relying on the compression algorithm.This may be done, for example, using intelligently filtered image tuplesof an image collection and an evolutionary algorithm. Details of thetypes of evolutionary algorithms which may be used in variousembodiments are provided in FIG. 14 and discussed below. In someimplementations, adversarial image tuples of the kind discussed earliermay be identified and included in the data set used to tune the PQalgorithms.

Once the PQ algorithms have been independently tuned, they may beutilized to generate PQ scores (predictions of perceived degradation inquality of compressed versions of images, relative to reference images)for an image data set representative of the image presentation context(element 1304). The compressed versions of the images for which the PQscores are generated may be obtained using various combinations ofhyper-parameter values of the compression algorithm. An evolutionaryalgorithm may also be used for tuning the compression algorithm invarious embodiments, with independent evolutionary algorithm parametersbeing used for tuning the compression algorithm than were used fortuning the PQ algorithms.

A tuned compression algorithm, obtained as the output of operationscorresponding to element 1304, may then be evaluated using a set ofhuman-annotated images which were not used earlier (for tuning eitherthe PQ algorithms or the compression algorithm) in the depictedembodiment (element 1307). In addition, in at least some embodiments,statistical tests may be run as part of the evaluation phase tocompensate for potential biases which might otherwise have reduced theapparent variance in the evaluation results.

If the quality of the tuned compression algorithm is found to besatisfactory in the evaluation phase (as determined in operationscorresponding to element 1310), the optimization procedure may beterminated, and the tuned compression algorithm may be deployed for theimage presentation context for which it was tuned (element 1316). If thequality of the compression algorithm is found unsatisfactory (as alsodetected in operations corresponding to element 1310), anotheroptimization iteration may be started (element 1313). Optionally, someof the newly-labeled images used in the evaluation phase of the currentiteration may be re-used in the PQ algorithm tuning phase of the nextiteration.

Overview of Evolutionary Algorithms

Evolutionary or genetics-based algorithms, which attempt to mimicaspects of individual-to-individual variation and fitness-dependentsurvival phenomena observed in biological species, may be used to tunePQ algorithms as well as compression algorithms in at least someembodiments as indicated above. Such algorithms may be especiallybeneficial for use in the compression context because, given theextremely large number of possible hyper-parameter value combinations,performing exhaustive or grid-based optimization may be computationallyinfeasible. FIG. 14 is a flow diagram illustrating a high-level overviewof an evolutionary algorithm which may be employed at one or more stagesof an iterative compression algorithm optimization technique, accordingto at least some embodiments.

As shown in element 1401, a fitness function and an initial populationof individuals may be selected for a given optimization problem beingaddressed using the evolutionary approach. Each of the individuals mayrepresent a hyper-parameter combination (PQ algorithm hyper-parametersfor the first stage of the optimization iterations discussed in thecontext of FIG. 13, and compression algorithm hyper-parameters for thesecond stage), selected for the particular image presentation contextfor which optimization is being performed. The fitness function may beused to select, from a given group of individuals of one generation oriteration of the evolutionary algorithm, the ones that survive for thenext generation (either in unmodified form, or with some variations).For example, for PQ algorithm optimization, in some embodiments thefitness function may indicate how closely the quality degradationpredictions obtained with a given PQ hyper-parameter combination matchthe judgments of human annotators. For compression algorithms, in atleast some embodiments the fitness function may be based on acombination of the file sizes of the compressed images, and the PQscores generated for the compressed images. In some embodiments, aknowledge base of a media optimization service may be used to select theinitial population and/or the fitness function for a given imagepresentation context or use case; the knowledge base may, for example,contain entries indicating hyper-parameter value combinations which haveworked well for similar contexts in the past, and such combinations maybe used for the initial population.

Using genetics-inspired techniques such as cross-over (copying andoverwriting subsets of hyper-parameters from one individual to anotherand mutation (e.g., introducing random perturbations in hyper-parametervalues), a set of child individuals may be created from the inputpopulation (the “parents”) of a given generation (element 1404). (Forthe very first generation, the initial population is the inputpopulation.) Cross-over and mutation may respectively representexploitation and exploration aspects of optimization: cross-over tendsto retain characteristics of “fitter” individuals across generations andthus represents exploitation of known successful characteristics, whilemutation introduces new variants which could potentially lead to thediscovery and exploration of new successful characteristics. Theexploitation versus exploration tradeoff may also be reflected byparameters which determine how many children are to be created in agiven generation from a given number of parents. A set of parametersrepresenting exploitation versus exploration tradeoffs (e.g., the numberof children per parent, the rate/number of mutations and/or therate/number/size of cross-over sets of hyper-parameters) may be chosenfor the evolutionary algorithm in various embodiments, e.g., by an MOSor by a client on whose behalf the MOS is performing evolutionaryoptimization iterations. Results of the fitness function may be obtainedfor the individuals of the input population and their children (element1407).

Evolutionary algorithms may be executed until a pre-selected limit onthe number of generations has been reached in various embodiments. Inoperations corresponding to element 1410, a determination may be made asto whether this limit has been reached in the depicted embodiment. Ifthe limit has been reached, the evolutionary procedure may be terminated(element 1416). One or more hyper-parameter combinations that had thebest fitness scores found thus far may be chosen for deployment or forfurther analysis in various embodiments. If the generation count limithas not been reached, as also determined in operations corresponding toelement 1410, the fitness function results or scores may be used toprune some fraction of the less-fit individuals, and the remainingindividuals (which may in principle include some children and someparents, only children, or only parents) may form the input populationof the next generation. Operations corresponding to elements 1404onwards may then be performed with respect to the next generation. It isnoted that other variants of evolutionary techniques may be used in someembodiments than that shown in FIG. 14: for example, less-fit parentsmay be pruned before any children are generated using cross-over ormutation in one embodiment.

Methods for Iterative Evolutionary Optimization of CompressionAlgorithms

FIG. 15 is a flow diagram illustrating aspects of an iterativecompression algorithm optimization technique in which evolutionarytechniques are employed, according to at least some embodiments. Asshown in element 1501, information about a target image presentationcontext (IPC) or use case may be obtained, for example at a mediaoptimization service (MOS) similar in functionality to MOS 110 ofFIG. 1. IPCs may differ from one another in various properties, such asthe image file format being used (which in turn may limit the kinds ofcompression algorithms which can be used), the kinds of interfacedevices (where the devices are characterized by both their hardware andsoftware components, including browsers and the like) expected to beused by viewers to view the images, the latency constraints with respectto compression (e.g., how quickly a compressed version of an image hasto be presented after the original image is received or selected forpresentation), and so on. IPC characteristics (including interfacedevices used by viewers, latency constraints, etc.) may be used toselect at least some hyper-parameter values to be explored for thecompression algorithms being optimized in different embodiments.

A particular PQ algorithm PQA-1 may be selected for optimizing aparticular image compression algorithm ICA-1 for the IPC in the depictedembodiment (element 1504), along with a representative image collectionor RIC. In some embodiments, one or both of the algorithms and/or an RICmay be indicated by a client of an MOS via programmatic interfaces. Inother embodiments, the MOS may select one or both algorithms and/or theRIC based on the IPC information provided by the client (or based onanalysis of webserver logs provided by the client). The sets ofhyper-parameters to be tuned for PQA-1 and ICA-1 may be identified, anda reference image compression algorithm ICA-R (with which thecompression applied using various hyper-parameter combinations of ICA-1are to be compared for optimization purposes) may be selected.

Having identified the algorithms and hyper-parameters, one or moremulti-stage optimization iterations may be initiated in variousembodiments (as indicated in element 1507). In the first stage,hyper-parameters of PQA-1 may be tuned (element 1510). A plurality ofimage tuples or image pairs in the image file format of the IPC may bechosen, and an evolutionary algorithm with a first set of explorationversus exploitation parameters (e.g., children-per-parent, cross-overrates, mutation rates, etc., as discussed above in the context of FIG.14) may be used to tune the PQA-1 hyper-parameters in the depictedembodiment. In at least some embodiments, the fitness function used foroptimizing PQA-1 may be based on measures of disagreement betweenintra-image-pair quality preferences indicated by human annotators andintra-image-pair quality preference predicted/generated by PQA-1. In atleast some embodiments, one or more image labels that were generated inthe evaluation stage (comprising operations corresponding to element1519 of FIG. 15) of the previous optimization iteration may be used inthe tuning of PQA-1.

In one implementation, for example, the following formulae describe azero-one loss optimization PQA-1 hyper-parameters θ. Consider a datasetD, comprised of some number n of i tuples (x, y). x is itself an imagetriple x={x₁, x₂, x₃} in which x₁ and x₂are images produced usingrespective ICA hyper-parameter combinations being compared and x₃ is areference image (e.g., the uncompressed version, or a version compressedusing a reference ICA such as ICA-R). y ∈{0,1} is the human-generatedannotation of whether x1 or x2 is preferred (e.g., with 0 indicatingthat x1 is preferred over x2 when comparing both x1 and x2 to x3, and 1indicating that x2 is preferred over x1). The evolutionary algorithmthen seeks the PQA-1 parameters that minimize a zero-one loss function

$\overset{\hat{}}{\theta} = {\arg\mspace{11mu}{\min\limits_{\theta}{\sum\limits_{i = 1}^{n}{L\left( {y_{i},\ {f\left( {x_{i};\theta} \right)}} \right)}}}}$

where L is the 0-1 loss, i.e.

${L\left( {i,j} \right)} = \left\{ {\begin{matrix}{0,} & {if} & {{i = j},} \\{1,} & {else} & \;\end{matrix}.} \right.$

and ƒ (x_(i); θ) is a function of PQA-1, which calculates or predictswhich image among x1 and x2 is preferred under hyper-parameters θ.

In one embodiment in which the MS-SIM algorithm introduced earlier isused as PQA-1, the hyper-parameters optimized using the above formulaemay include, for example, the luminance, contrast and similarityexponents α_(M), β_(M) and γ_(M). In an embodiment in which the fusionmodel-based algorithm is chosen as PQA-1, the hyper-parameters optimizedusing the above formulae may include, for example, the number of PQalgorithms whose scores are used in the training records, the types ofPQ algorithms whose scores are used in the training records, whetherfinal scores of the PQ algorithms are used or internal intermediaryscores of the PQ algorithms are used, the number of fully-connectedlayers in the neural network of the fusion model, the number of “fake”compressed images (which are actually the reference images) to be usedfor the training records, and so on.

In the embodiment depicted in FIG. 15, the second stage of theoptimization iteration may comprise using the tuned version of PQA-1identified in the same iteration to tune ICA-1 (element 1510). Usingthis version of PQA-1, PQ scores may be generated for another set ofimage pair or tuples (different from those used for optimizing PQA-1) ofthe RIC. A given image pair may contain one image compressed using aparticular combination of hyper-parameter values of ICA-1, and one imagecompressed using the reference ICA, ICA-R. An evolutionary algorithmwith a different fitness function and a differentexploitation-exploration parameters may be used to identify thebest-performing hyper-parameter combinations of ICA-1 in the depictedembodiment. The fitness function may be based on the size of theICA-1-compressed file size (with smaller file sizes indicating superiorfitness), as well as on the PQ scores generated using PQA-1 (withsmaller differences with respect to the reference image indicatingsuperior fitness).

In one implementation, logic similar to the following may be used tocompute penalties associated with compressed file sizes, and thepenalties thus computed may be used in the fitness function.

if PQ_degradation[compressed_image relative to reference image]>1.02then penalty[compressed_image]+=(50*file_size[compressed_image]);

if PQ_degradation[compressed_image relative to reference image]>1.05then penalty[compressed_image]+=(100*file_size[compressed_image]);

In the above if statements, higher PQ_degredation values (1.05 relativeto 1.02) correspond to predictions of worse perceived quality—that is,the PQ algorithm predicts that a compressed image with the 1.05degradation score would be considered worse, relative to the referenceimage, than a compressed image with a degradation score of 1.02. Thepenalty for the compressed image increases with the file size, and alsoincreases with increases in predicted perceived quality degradation. Theapproach represented by the logic shown above has the advantage that ifa given combination HPC1 of hyper-parameters results in a worsedegradation score than a different combination HPC2, HPC1 could still beconsidered a “fitter” combination (or an equally fit combination)relative to HPC2 if HPC1 results in a smaller compressed file size thanHPC2. As such, this approach attempts to strike a balance between thegoals of obtaining high-perceived-quality compressed images and reducingcompressed image file size (and does not reject compressionhyper-parameters based purely on file size alone or perceived qualityalone).

After the fittest hyper-parameter combination for ICA-1 is identified, athird stage of the optimization iteration may be conducted, comprisingevaluation of the tuned version of ICA-1 in the depicted embodiment(element 1519). In this stage, previously-unused human-annotated imagesas well as statistical tests which adjust for potential biases (e.g.,annotator-specific biases which may tend to make some annotators prefera particular image of a pair based on the relative locations of theimages of the pair on a display screen, or biases which areimage-specific) may be used to ensure that the tuned version of ICA-1 isevaluated fairly. The biases may be modeled as random effects, e.g.,using a logistic mixed-effects statistical model in some embodiments,which has the effect of increasing the variance associated with theevaluation results. In the operations corresponding to element 1519, adetermination may be made as to how similar in quality the compressedimages generated by the tuned version of ICA-1 are to the compressedimages generated by ICA-R (at least some of which may be larger in filesize), in the opinion of the human annotators.

If the results of the evaluation indicate that images generated by thetuned version of ICA-1 is deemed sufficiently similar to the referencecompressed images, as detected in operations corresponding to element1522, the optimization iterations may be terminated (element 1525). Thetuned hyper-parameter values of ICA-1 may be stored in variousembodiments, e.g., in a results repository of the media optimizationservice. A tuned version of ICA-1, obtained for example from the finaloptimization iteration, may be utilized for generating images presentedin the IPC. If further iterations are needed, operations correspondingto elements 1507 onwards may be conducted for the next iteration. Someof the human-annotated images generated in the evaluation stage of thecurrent iteration may be re-used for further tuning of PQA-1 in the nextiteration in at least some embodiments, as mentioned above.

Example Programmatic Interactions Pertaining to Compression Optimization

FIG. 16 illustrates example programmatic interactions between a clientand a media optimization service, pertaining to iterative compressionalgorithm optimization, according to at least some embodiments.Programmatic interfaces 1677, which may include APIs, web-basedconsoles, command-line tools, and/or graphical user interfaces may beimplemented by an MOS 1612 which supports iterative compressionalgorithm optimization techniques similar to those discussed in thecontext of FIG. 13-FIG. 15. Using such interfaces, a client 1610 maysend an ImagePresentationContextsInfo message 1614, indicating variousproperties of one or more target image presentation contexts (IPCs) forwhich compression algorithms are to be optimized. The information mayindicate the kinds of file formats to be used for compressed images, thetypes of interfaces/devices to be used by image viewers, resource andtiming constraints of the image presentation pipeline (e.g., time limitsbetween the decision to present a given image and the time that acompressed version of the image is to be shown), resource constraints ofthe end-user devices (such as computing capacity limits, memoryconstraints, etc.) and so on. Resource constraints of the imagepresentation context and/or he interfaces/devices of the viewers may beused, for example, to select initial hyper-parameter values ofcompression algorithms during the optimization iterations. The MOS 1612may store the provided information, and send an IPCInfoSaved responsemessage 1615 back to the client in some embodiments.

In some embodiments, a client 1610 may submit anIdentifyPQAlgorithmsForIPC request, indicating that one or more PQalgorithms suitable for a specified IPC be selected to tune one or morecompression algorithms. In response, a knowledge base or a set ofcompression experts may be consulted, and an indication of a recommendedPQ algorithm (such as the MS-SIM, HDR-VDP2, HaarPSI, or fusionalgorithms discussed earlier) may be provided to the client via aPQAlgorithmSelected message 1621.

A client may submit an IdentifyCompressionAlgorithmsForIPC request 1623in various embodiments to the MOS 1612, requesting that candidatecompression algorithms (CAs) appropriate for a specified IPC beidentified. In response, as in the case of the PQ algorithms, aknowledge base and/or a set of compression experts may be consulted bythe MOS, and an indication of a recommended compression algorithm (e.g.,some variant of a JPEG compression algorithm) may be provided to theclient in a CAldentified message 1625. Note that in some embodiments,clients may choose PQ algorithms and/or candidate compression algorithmson their own, and may simply notify the MOS 1612 programmatically aboutthe algorithms to be used instead of sending requests similar to 1617 or1623.

In at least one embodiment, a client 1610 may submit an indication ofpreferences for iterative compression algorithm tuning, e.g., via one ormore PQAandCATuningPreferences messages 1628. These messages mayindicate, for example, recommendations for identifying image tuples tobe used, initial combinations of hyper-parameters to be used forstarting the tuning of a PQ algorithms and/or a compression algorithms(which may be used for the initial populations of the evolutionaryalgorithms), suggested values or ranges for evolutionary algorithmparameters (including parameters representing tradeoffs betweenexploration and exploitation discussed above), the maximum count ofevolutionary algorithm generations for tuning PQ algorithms andcompression algorithms, etc. In effect, the tuning preferences indicatedby a client via a message 1628 may help the MOS select values for higherlevel hyper-parameter for the iterative tuning of the PQA/CAhyper-parameters. A message 1633 indicating that the tuning preferenceshave been saved may be sent to the client from the MOS in someembodiments after the tuning preferences have been stored.

A client 1610 may submit an InitiatePQAandCATuning request 1641 in someembodiments to trigger the execution of optimization iterations similarto those discussed in the contexts of FIG. 15 or FIG. 13. In someembodiments, a client may not necessarily submit requests to identify PQalgorithms (e.g., similar to requests 1617) and/or compressionalgorithms (e.g., similar to request 1619) prior to submitting such atuning request. In one such embodiment, the tuning request 1641 mayindicate the PQ and compression algorithms to be used. In otherembodiments, the MOS 1612 may select appropriate PQ and/or compressionalgorithms without being separately requested to do so. Multi-stageoptimization iterations may be initiated at the MOS in response to atuning request 1641, and a TuninglterationsStarted message 1643 may besent to the client in the depicted embodiment. In some embodiments, asindividual iterations are completed, results obtained from theiterations (e.g., quality measures or fitness scores) may be provided tothe client 1610 in one or more IntermediateResults messages 1645. Afterthe optimization procedure is terminated, a TuningCompleted message 1647may be sent to the client. A client 1610 may request that the tunedcompression algorithm be deployed for one or more image presentationcontexts, e.g., by a content presentation service, by sending aDeployTunedCompressionAlgorithm message 1651 to the MOS 1612 in someembodiments. After the algorithm has been deployed, an AlgorithmDeployedmessage acknowledging the deployment may be sent to the client.

Example Class-Specific Tuning of Compression Algorithms

The techniques for enhancing perceived quality predictions and/oriteratively optimizing compression algorithms may be employed in manydifferent ways in different embodiments. FIG. 17 is a flow diagramillustrating aspects of a technique in which image class-specific tuningof compression algorithms may be conducted, according to at least someembodiments. As shown in element 1701, a compression algorithm may betuned using a representative set of images presented in a given context(such as from an e-retail web site to customers browsing or orderingitems from laptops/tablet devices/phones). The iterative evolutionarytechnique discussed earlier may be used in some embodiments. Thehyper-parameter values which lead to the least degradation in qualitymay be identified as part of the tuning exercise, and stored in adefault hyper-parameter set HS-default.

One or more important classes of images being served in the givencontext may be identified, e.g., using machine learning-based objectrecognition and/or clustering algorithms in the depicted embodiment(element 1704). The definition of “important” may vary for differentclients of a media optimization service: for example, images that leadto more revenue being generated overall for an e-retail enterprise maybe considered important, or images that lead to more revenue generatedper image view may be considered more important. In one e-retailscenario, for example, important class Cl may comprise images ofelectronic appliances, important class C2 may comprise watches,important class C3 may comprise shoes, etc. In at least someembodiments, clients may programmatically indicate the factors whichcontribute to importance for their applications, or provide examples ofimages which can be used (e.g., using searches for image with similarsubjects) to classify some images as more important than others.

For each important class identified, a separate compressionhyper-parameter tuning exercise may be conducted in at least someembodiments (element 1707), e.g., again using the evolutionary approachdiscussed earlier. Class-specific tuned hyper-parameter valuecombinations HS-C1 for class C1, HS-C2 for class C2, etc., may beidentified using this approach. The HA-default and class-specific tunedhyper-parameter values may be provided to content presentation servicesbeing used for presenting the images to viewers in the depictedembodiment.

When a given image is to be presented to a viewer, a determination maybe made at the content presentation service whether the image belongs toone of the important classes identified earlier (element 1710). If so,the appropriate class-specific hyper-parameter combination may beemployed to compress the image prior to transmission of the image to themedia consumption interface device of the viewer in the depictedembodiment. If the image does not belong to one of the importantclasses, the HS-default values may be used. In at least someimplementations, the process of presenting the images may include thetransmission of class-specific data (e.g., tuned class-specificquantization tables) to be used in the de-compression at the mediaconsumption interface devices.

Example Use of Sub-Image-Specific Compression Algorithms

FIG. 18 is a flow diagram illustrating aspects of a technique in whichcustom tuning of compression algorithms may be conducted for sub-imagesof a given image, according to at least some embodiments. As shown inelement 1801, a set of images S1 may be identified for which very highperceived-quality compressed versions are to be presented to viewers.For some websites, visitors to the website may typically access thewebsite through a home page or a small number of landing pages; as aresult, the images shown on such pages may be viewed by all the visitorsto the web site and may be designated as requiring very high qualitycompressed versions.

The images of set S1 may be divided into respective groups of sub-imagesin the depicted embodiment (element 1804). This subdivision may beperformed using a variety of techniques in different embodiments. Insome embodiments, the image may simply be divided into blocks of pixels(e.g., 32×32 pixel blocks, 128×128 pixel blocks, etc.). In otherembodiments, the images may be subdivided based on image contentinterpretation or semantics, with one sub-image being identified for aforeground face, another for furniture in the background, and so on.

Hyper-parameters for a compression algorithm may then be tunedseparately for individual sub-images in the depicted embodiment (element1807), e.g., using distinct sets of evolutionary iterations of the kinddiscussed earlier. The separately-tuned hyper-parameter combinations maybe used to present the image as a whole, applying different combinationsof hyper-parameters to each sub-image (element 1810).

It is noted that in various embodiments, some of the operations shown inthe flow diagrams of FIG. 8, FIG. 9, FIG. 13, FIG. 14, FIG. 15, FIG. 17,and/or FIG. 18 may be implemented in a different order than that shownin the figure, or may be performed in parallel rather than sequentially.Additionally, some of the operations shown in FIG. 8, FIG. 9, FIG. 13,FIG. 14, FIG. 15, FIG. 17, and/or FIG. 18 may not be required in one ormore implementations.

Use Cases

The techniques described above, of training a machine learning model forperceptual quality prediction using scores obtained from multiple PQalgorithms for adversarial image tuples, and the iterative multi-stageoptimization of compression algorithms, may be extremely beneficial in anumber of environments. E-retail organizations may have to servemillions of images and/or other content items every day, and even asmall reduction in the sizes of the media objects may result insubstantial reductions in network bandwidth used, as well as quickerresponses to requests from clients. Services or organizations providingaccess to e-books, e-magazines, e-comics and the like may also benefitfrom the proposed techniques.

Illustrative Computer System

In at least some embodiments, a server that implements one or more ofthe techniques described herein (e.g., training and executing enhancedmodels for perceptual quality predictions, optimizing compressionalgorithms, and/or other functions of a media optimization serviceand/or a content presentation service), may include a general-purposecomputer system that includes or is configured to access one or morecomputer-accessible media. FIG. 19 illustrates such a general-purposecomputing device 9000. In the illustrated embodiment, computing device9000 includes one or more processors 9010 coupled to a system memory9020 (which may comprise both non-volatile and volatile memory modules)via an input/output (I/O) interface 9030. Computing device 9000 furtherincludes a network interface 9040 coupled to I/O interface 9030.

In various embodiments, computing device 9000 may be a uniprocessorsystem including one processor 9010, or a multiprocessor systemincluding several processors 9010 (e.g., two, four, eight, or anothersuitable number). Processors 9010 may be any suitable processors capableof executing instructions. For example, in various embodiments,processors 9010 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 9010 may commonly,but not necessarily, implement the same ISA. In some implementations,graphics processing units (GPUs) may be used instead of, or in additionto, conventional processors.

System memory 9020 may be configured to store instructions and dataaccessible by processor(s) 9010. In at least some embodiments, thesystem memory 9020 may comprise both volatile and non-volatile portions;in other embodiments, only volatile memory may be used. In variousembodiments, the volatile portion of system memory 9020 may beimplemented using any suitable memory technology, such as static randomaccess memory (SRAM), synchronous dynamic RAM or any other type ofmemory. For the non-volatile portion of system memory (which maycomprise one or more NVDIMMs, for example), in some embodimentsflash-based memory devices, including NAND-flash devices, may be used.In at least some embodiments, the non-volatile portion of the systemmemory may include a power source, such as a supercapacitor or otherpower storage device (e.g., a battery). In various embodiments,memristor based resistive random access memory (ReRAM),three-dimensional NAND technologies, Ferroelectric RAM, magnetoresistiveRAM (MRAM), or any of various types of phase change memory (PCM) may beused at least for the non-volatile portion of system memory. In theillustrated embodiment, program instructions and data implementing oneor more desired functions, such as those methods, techniques, and datadescribed above, are shown stored within system memory 9020 as code 9025and data 9026.

In one embodiment, I/O interface 9030 may be configured to coordinateI/O traffic between processor 9010, system memory 9020, and anyperipheral devices in the device, including network interface 9040 orother peripheral interfaces such as various types of persistent and/orvolatile storage devices. In some embodiments, I/O interface 9030 mayperform any necessary protocol, timing or other data transformations toconvert data signals from one component (e.g., system memory 9020) intoa format suitable for use by another component (e.g., processor 9010).In some embodiments, I/O interface 9030 may include support for devicesattached through various types of peripheral buses, such as a variant ofthe Peripheral Component Interconnect (PCI) bus standard or theUniversal Serial Bus (USB) standard, for example. In some embodiments,the function of I/O interface 9030 may be split into two or moreseparate components, such as a north bridge and a south bridge, forexample. Also, in some embodiments some or all of the functionality ofI/O interface 9030, such as an interface to system memory 9020, may beincorporated directly into processor 9010.

Network interface 9040 may be configured to allow data to be exchangedbetween computing device 9000 and other devices 9060 attached to anetwork or networks 9050, such as other computer systems or devices asillustrated in FIG. 1 through FIG. 18, for example. In variousembodiments, network interface 9040 may support communication via anysuitable wired or wireless general data networks, such as types ofEthernet network, for example. Additionally, network interface 9040 maysupport communication via telecommunications/telephony networks such asanalog voice networks or digital fiber communications networks, viastorage area networks such as Fibre Channel SANs, or via any othersuitable type of network and/or protocol.

In some embodiments, system memory 9020 may represent one embodiment ofa computer-accessible medium configured to store at least a subset ofprogram instructions and data used for implementing the methods andapparatus discussed in the context of FIG. 1 through FIG. 18. However,in other embodiments, program instructions and/or data may be received,sent or stored upon different types of computer-accessible media.Generally speaking, a computer-accessible medium may includenon-transitory storage media or memory media such as magnetic or opticalmedia, e.g., disk or DVD/CD coupled to computing device 9000 via I/Ointerface 9030. A non-transitory computer-accessible storage medium mayalso include any volatile or non-volatile media such as RAM (e.g. SDRAM,DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., that may be included in someembodiments of computing device 9000 as system memory 9020 or anothertype of memory. In some embodiments, a plurality of non-transitorycomputer-readable storage media may collectively store programinstructions that when executed on or across one or more processorsimplement at least a subset of the methods and techniques describedabove. A computer-accessible medium may further include transmissionmedia or signals such as electrical, electromagnetic, or digitalsignals, conveyed via a communication medium such as a network and/or awireless link, such as may be implemented via network interface 9040.Portions or all of multiple computing devices such as that illustratedin FIG. 19 may be used to implement the described functionality invarious embodiments; for example, software components running on avariety of different devices and servers may collaborate to provide thefunctionality. In some embodiments, portions of the describedfunctionality may be implemented using storage devices, network devices,or special-purpose computer systems, in addition to or instead of beingimplemented using general-purpose computer systems. The term “computingdevice”, as used herein, refers to at least all these types of devices,and is not limited to these types of devices.

Embodiments of the disclosure can be described in view of the followingclauses:

Clause 1. A system, comprising:

-   -   one or more computing devices;    -   wherein the one or more computing devices include instructions        that upon execution on or across the one or more computing        devices cause the one or more computing devices to:        -   identify a first plurality of image tuples which satisfy an            algorithm-to-algorithm divergence threshold, wherein            individual ones of the image tuples comprise a reference            image, a first compressed version of the reference image,            and a second compressed version of the reference image, and            wherein, with respect to a given image tuple, a difference            between (a) a first quality degradation score produced by a            first perceptual quality algorithm of a first set of            perceptual quality algorithms for one or more of the            compressed versions relative to the reference image, and (b)            a second quality degradation score produced by a second            perceptual quality algorithm of the first set for the one or            more of the compressed versions relative to the reference            image exceeds the divergence threshold;        -   obtain respective labels from a group of one or more            annotators for individual ones of the first plurality of            image tuples, wherein a label for a given image tuple            indicates which compressed version of the given image tuple            is perceived to be more similar to the reference image of            the given image tuple;        -   without utilizing an annotator, automatically generate            labels for individual ones of a second plurality of image            tuples using quality degradation scores produced by a second            set of perceptual quality algorithms;        -   store a labeled image data set comprising at least some            image tuples of the first and second pluralities of image            tuples and their respective labels;        -   generate, using a third set of perceptual quality            algorithms, a plurality of pairs of training records for at            least a first machine learning model, wherein an individual            pair of training records comprises:            -   a first record which includes (a) a plurality of quality                degradation scores for a first compressed image of the                labeled image data set, wherein individual ones of the                quality degradation scores are obtained using respective                perceptual quality algorithms of the third set, and (b)                the particular label which was stored in the labeled                image data set for the image tuple of which the first                compressed version is a member; and            -   a second record which includes (a) a plurality of                quality degradation scores for a second compressed image                of the labeled image data set, wherein individual ones                of the quality degradation scores are obtained using the                respective perceptual quality algorithms of the third                set, and (b) the particular label;        -   train the first machine learning model using the plurality            of pairs of training records to predict, for a post-training            input record comprising a plurality of quality degradation            scores for a particular compressed version of an image, an            output quality degradation score for the particular            compressed version; and        -   utilize the output quality degradation score to identify an            image for presentation to a viewer.

Clause 2. The system as recited in clause 1, wherein the first machinelearning model comprises a symmetric neural network with at least onefully-connected layer and at least one softmax layer.

Clause 3. The system as recited in any one of clauses 1 to 2, whereinthe one or more annotators comprise a plurality of annotators, andwherein the one or more computing devices include further instructionsthat upon execution on or across the one or more computing devicesfurther cause the one or more computing devices to:

-   -   analyze the inter-annotator consistency of labels produced by        the plurality of annotators; and    -   exclude, from the labeled image data set, at least one image        tuple based at least on part on results of the inter-annotator        consistency analysis.

Clause 4. The system as recited in any one of clauses 1 to 3, whereinthe one or more computing devices include further instructions that uponexecution on or across the one or more computing devices further causethe one or more computing devices to:

-   -   initiate the identification of the first plurality of image        tuples in response to one or more requests obtained via a        programmatic interface of a network-accessible service of a        provider network.

Clause 5. The system as recited in any one of clauses 1 to 4, wherein afirst the third set of perceptual quality algorithms comprises aparticular algorithm in which a final quality degradation score isobtained from a plurality of intermediate scores, wherein the pluralityof quality degradation scores included for the first compressed image inthe first training record comprises an intermediate score of theplurality of intermediate scores.

Clause 6. A method, comprising:

-   -   performing, at one or more computing devices:        -   obtaining respective labels from a group of one or more            annotators for individual ones of the first plurality of            image tuples which satisfy a first divergence criterion,            wherein individual ones of the image tuples comprise a            reference image, a first compressed version of the reference            image, and a second compressed version of the reference            image, and wherein a label for a given image tuple indicates            which compressed version of the given image tuple is            perceived to be more similar to the reference image of the            given image tuple;        -   storing an labeled image data set comprising at least some            image tuples of the first plurality of image tuples and            their respective labels;        -   generating, using a first set of perceptual quality            algorithms, a plurality of pairs of training records for at            least a first machine learning model, wherein an individual            pair of training records comprises:            -   a first record which includes a plurality of quality                degradation scores for a first compressed version of a                particular reference image of the labeled image data                set, wherein individual ones of the quality degradation                scores are obtained using respective perceptual quality                algorithms of the first set; and            -   a second record which includes a plurality of quality                degradation scores for a second compressed version of                the particular reference image, wherein individual ones                of the quality degradation scores are obtained using the                respective perceptual quality algorithms of the first                set; and        -   training the first machine learning model using the            plurality of pairs of training records to predict, for a            post-training input record comprising a plurality of quality            degradation scores for a particular compressed version of an            image, a quality degradation score for the particular            compressed version.

Clause 7. The method as recited in clause 6, further comprisingperforming, at one or more computing devices:

-   -   without utilizing an annotator, automatically generating labels        for individual ones of a second plurality of image tuples using        quality degradation scores generated by a second set of        perceptual quality algorithms; and    -   storing the second plurality of image tuples and their        respective labels as part of the labeled image data set.

Clause 8. The method as recited in any one of clauses 6 to 7, furthercomprising performing, at one or more computing devices:

-   -   determining a difference, with respect to a particular image        tuple of a collection of image tuples, between (a) a first        quality degradation score generated by a first perceptual        quality algorithm for a compressed image of the image tuple        and (b) a second quality degradation score generated by a second        perceptual quality algorithm for the compressed image; and    -   evaluating the first divergence criterion with respect to the        particular image tuple, wherein said evaluating comprises        comparing the difference to a threshold.

Clause 9. The method as recited in any one of clauses 6 to 8, whereinthe group of one or more annotators comprises a plurality of annotators,the method further comprising performing, at one or more computingdevices:

computing, for individual image tuples of the first plurality of imagetuples, a measure of inter-annotator consistency; and

-   -   excluding, from the labeled image data set, at least one image        tuple whose inter-annotator consistency measure is below a        threshold.

Clause 10. The method as recited in any one of clauses 6 to 9, whereinthe group of one or more annotators comprises a plurality of annotators,the method further comprising performing, at one or more computingdevices:

-   -   excluding, from the labeled image data set, at least one image        tuple for which a label was generated by a particular annotator        selected based on an analysis of inter-annotator consistency.

Clause 11. The method as recited in any one of clauses 6 to 10, furthercomprising performing, at one or more computing devices:

-   -   training, using additional training records for which labels        were generated automatically without using annotators, a second        machine learning model to predict quality degradation scores,        wherein training the first machine learning model using the        plurality of pairs of training records comprises modifying the        second machine learning model using the plurality of pairs of        training records.

Clause 12. The method as recited in any one of clauses 6 to 11, whereinthe first machine learning model comprises a neural network-based model.

Clause 13. The method as recited in any one of clauses 6 to 12, furthercomprising:

-   -   obtaining, at a network-accessible service of a provider        network, one or more programmatic requests to train a machine        learning model to predict perceived image quality degradation        scores, wherein the first machine learning model is trained in        response to the one or more programmatic requests.

Clause 14. The method as recited in any one of clauses 6 to 13, furthercomprising:

-   -   obtaining, from a trained version of the first machine learning        model, a first set of quality degradation scores for compressed        images produced using a first set of hyper-parameters of a        compression algorithm, and a second set of quality degradation        scores for compressed images produced using a second set of        hyper-parameters of the compression algorithm; and    -   causing, based at least in part on a comparison of the first and        second sets of quality degradation scores, the first set of        hyper-parameters to be employed for presenting a set of images.

Clause 15. The method as recited in any one of clauses 6 to 14, furthercomprising:

-   -   obtaining respective resource consumption metrics of a plurality        of perceptual quality algorithms; and    -   including, in the first set of perceptual quality algorithms, a        first perceptual quality algorithm of the plurality of        perceptual quality algorithms based at least in part on a        comparison of a resource consumption metric of the first        perceptual quality algorithm with a corresponding resource        consumption metric of a second perceptual quality algorithm of        the plurality of perceptual quality algorithms; and    -   excluding, from the first set of perceptual quality algorithms,        the second perceptual quality algorithm based at least in part        on the comparison.

Clause 16. One or more non-transitory computer-accessible storage mediastoring program instructions that when executed on or across one or moreprocessors cause the one or more processors to:

-   -   obtain respective labels from a group of one or more annotators        for individual ones of the first plurality of media object        tuples which satisfy a first divergence criterion, wherein        individual ones of the media object tuples comprise a reference        media object, a first compressed version of the media object,        and a second compressed version of the reference media object,        and wherein a label for a given media object tuple indicates        which compressed version of the given media object tuple is        perceived to be more similar to the reference media object of        the given media object tuple;    -   generate, using a first set of perceptual quality algorithms, a        plurality of pairs of training records for at least a first        machine learning model, wherein an individual pair of training        records comprises:        -   a first record which includes a plurality of quality            degradation scores for a first compressed version of a            particular reference media object of a labeled media object            data set, wherein individual ones of the quality degradation            scores are obtained using respective perceptual quality            algorithms of the first set, and wherein the labeled media            object data set comprises at least some media object tuples            of the first plurality of media object tuples and their            respective labels; and        -   a second record which includes a plurality of quality            degradation scores for a second compressed version of the            particular reference media object, wherein individual ones            of the quality degradation scores are obtained using the            respective perceptual quality algorithms of the first set;            and    -   train the first machine learning model using the plurality of        pairs of training records to predict, for a post-training input        record comprising a plurality of quality degradation scores for        a particular compressed version of a media object, a quality        degradation score for the particular compressed version.

Clause 17. The one or more non-transitory computer-accessible storagemedia as recited in clause 16, storing further program instructions thatwhen executed on or across the one or more processors further cause theone or more processors to:

-   -   automatically generate labels for individual ones of a second        plurality of media object tuples using quality degradation        scores generated by a second set of perceptual quality        algorithms; and    -   include the second plurality of media object tuples and their        respective labels as part of the labeled media object data set.

Clause 18. The one or more non-transitory computer-accessible storagemedia as recited in any one of clauses 16 to 17, wherein the first setof perceptual quality algorithms comprises one or more of: (a) analgorithm which utilizes multi-scale decomposition to generate predictedperceived quality degradation scores, (b) an algorithm in which physicalimage differences are weighted at least according to assumptions aboutcontrast sensitivity, or (c) an algorithm which measures phase coherencein spatial filters.

Clause 19. The one or more non-transitory computer-accessible storagemedia as recited in any one of clauses 16 to 18, wherein the firstmachine learning model comprises a neural network-based model.

Clause 20. The one or more non-transitory computer-accessible storagemedia as recited in any one of clauses 16 to 19, storing further programinstructions that when executed on or across the one or more processorsfurther cause the one or more processors to:

-   -   identify, using a second set of perceptual quality algorithms,        the media object tuples which satisfy the first divergence        criterion, wherein the at least one algorithm of the second set        is not in the first set of perceptual quality algorithms.

Clause 21. The one or more non-transitory computer-accessible storagemedia as recited in any one of clauses 16 to 20, storing further programinstructions that when executed on or across the one or more processorsfurther cause the one or more processors to:

-   -   perform one or more optimization iterations, wherein a        particular optimization iteration of the one or more        optimization iterations comprises at least:        -   tuning one or more hyper-parameters of a particular            perceptual quality algorithm which utilizes the first            machine learning model;        -   tuning, using a set of perceptual quality scores, one or            more hyper-parameters of a first media object compression            algorithm, wherein the set of perceptual quality scores is            generated for a first set of media objects using a tuned            version of the particular perceptual quality algorithm            obtained in the particular optimization iteration, and            wherein the set of media objects includes a compressed media            object obtained using a particular hyper-parameter            combination of the first media object compression algorithm;            and        -   conducting a compression quality evaluation test on a set of            media objects generated using a tuned version of the first            media object compression algorithm obtained in the            particular iteration, wherein a result of the quality            evaluation test is used to determine whether additional            optimization iterations are to be performed.

Clause 22. A system, comprising:

-   -   one or more computing devices;    -   wherein the one or more computing devices include instructions        that upon execution on or across the one or more computing        devices cause the one or more computing devices to:        -   obtain an indication, at a media optimization service, of            one or more properties of a first image presentation            context, including (a) a first image file format and (b) a            first type of interface used to present images to viewers;        -   identify, at the media optimization service, based at least            in part on the one or more properties, at least a first            perceptual quality algorithm to be used to optimize a first            image compression algorithm for the first image presentation            context;        -   perform, at the media optimization service, a plurality of            optimization iterations, wherein a particular optimization            iteration comprises at least:            -   tuning one or more hyper-parameters of the first                perceptual quality algorithm using an evolutionary                algorithm with a first fitness function and an annotated                input data set comprising a first plurality of image                pairs in the first image file format, wherein respective                annotations for at least some image pairs of the input                data set are generated in a compression quality                evaluation test of an earlier optimization iteration;            -   tuning, using a set of perceptual quality scores and an                evolutionary algorithm with a second fitness function,                one or more hyper-parameters of the first image                compression algorithm, wherein the set of perceptual                quality scores is generated for a second plurality of                image pairs using a tuned version of the first                perceptual quality algorithm obtained in the particular                optimization iteration, and wherein individual ones of                the second plurality of image pairs include (a) an image                compressed using a particular hyper-parameter                combination of the first image compression algorithm                and (b) an image compressed using a reference image                compression algorithm; and            -   conducting a compression quality evaluation test on a                set of images generated using a tuned version of the                first image compression algorithm obtained in the                particular iteration, wherein a result of the                compression quality evaluation test is used to determine                whether additional optimization iterations are to be                performed; and        -   cause to be presented, via one or more interfaces of the            first type of interface, one or more images encoded using a            tuned version of the first image compression algorithm            obtained from a final optimization iteration of the            plurality of optimization iterations.

Clause 23. The system as recited in clause 22, wherein input of thesecond fitness function with respect to the particular hyper-parametercombination comprises one or more of: (a) a perceptual quality scoreobtained for the image compressed using the particular hyper-parametercombination or (b) a size of the image compressed using the particularhyper-parameter combination.

Clause 24. The system as recited in any one of clauses 22 to 23, whereinthe one or more computing devices include further instructions that uponexecution on or across the one or more computing devices further causethe one or more computing devices to:

-   -   obtain a request via a programmatic request of the media        optimization service, wherein the plurality of optimization        iterations is initiated in response to the request.

Clause 25. The system as recited in any one of clauses 22 to 24, whereinthe first image file format comprises one of: (a) a JPEG (JointPhotographic Experts Group) format, (b) a WebP format, (c) an AVIFformat, or (d) an MP4 (Motion Pictures Experts Group—4) format.

Clause 26. The system as recited in any one of clauses 22 to 25, whereinthe one or more hyper-parameters of the first compression algorithmcomprise a parameter associated with one or more of: (a) chromasubsampling, (b) block prediction, (c) frequency domain transformation,(d) quantization or (e) run-length encoding.

Clause 27. A method, comprising:

-   -   performing, at one or more computing devices:        -   one or more optimization iterations, wherein a particular            optimization iteration of the one or more optimization            iterations comprises at least:            -   tuning one or more hyper-parameters of a first                perceptual quality algorithm;            -   tuning, using a set of perceptual quality scores, one or                more hyper-parameters of a first image compression                algorithm, wherein the set of perceptual quality scores                is generated for a first plurality of image pairs using                a tuned version of the first perceptual quality                algorithm obtained in the particular optimization                iteration, and wherein individual ones of the plurality                of image pairs include (a) an image compressed using a                particular hyper-parameter combination of a first image                compression algorithm and (b) an image compressed using                a reference compression algorithm; and            -   conducting a compression quality evaluation test on a                set of images generated using a tuned version of the                first image compression algorithm obtained in the                particular iteration, wherein a result of the quality                evaluation test is used to determine whether additional                optimization iterations are to be performed; and        -   storing tuned hyper-parameter values of the first            compression algorithm which were determined in the one or            more optimization iterations.

Clause 28. The method as recited in clause 27, wherein said tuning theone or more hyper-parameters of the first image compression algorithmcomprises utilizing an evolutionary algorithm.

Clause 29. The method as recited in clause 28, wherein a fitnessfunction used in the evolutionary algorithm is based at least in part ona penalty value, wherein the penalty value depends at least in part on afile size of a compressed file generated using the first imagecompression algorithm.

Clause 30. The method as recited in clause 29, wherein the penalty valuedepends on a difference in perceptual quality scores between (a) thecompressed file generated using the first image compression algorithmand (b) a compressed file generated using the reference compressionalgorithm.

Clause 31. The method as recited in clause 28, further comprisingperforming, at the one or more computing devices:

determining an initial combination of hyper-parameters of the firstimage compression algorithm for a first optimization iteration of theone or more optimization iterations based at least in part on one ormore of: (a) input received via a programmatic interface, or (b) aknowledge base.

Clause 32. The method as recited in any one of clauses 27 to 28, whereinsaid tuning the one or more hyper-parameters of the first perceptualquality algorithm comprises utilizing an evolutionary algorithm.

Clause 33. The method as recited in clause 32, wherein a fitnessfunction used in the evolutionary algorithm used for tuning the one ormore hyper-parameters of the first perceptual quality algorithm is basedat least in part on a metric of disagreement between (a)intra-image-pair quality preferences indicated by one or more annotatorsand (b) corresponding intra-image-pair quality preferences generated bythe perceptual quality algorithm.

Clause 34. The method as recited in any one of clauses 27, 28 or 32,wherein said conducting the compression quality evaluation testcomprises:

-   -   identifying a set of source images which have not been used        earlier in the one or more optimization iterations;    -   generating (a) a first compressed version a particular source        image of the set using the reference compression algorithm        and (a) a second compressed version of the particular source        image using the tuned version of the first compression        algorithm;    -   obtaining, from one or more annotators, respective indications        of the perceived similarity of the first and second compressed        versions to the particular source image; and    -   utilizing the respective indications to obtain an aggregate        metric of a difference between (a) the perceived similarity, to        the source images, of compressed versions of the source images        obtained using the tuned version of the first image compression        algorithm and (b) the perceived similarity, to the source        images, of compressed versions of the source images obtained        using the reference image compression algorithm.

Clause 35. The method as recited in clause 34, further comprising:

-   -   utilizing, as part of an input data set for the tuning of the        one or more hyper-parameters of the first compression algorithm        in a subsequent optimization iteration, the first compressed        version, the second compressed version and associated similarity        indications obtained from the one or more annotators.

Clause 36. The method as recited in clause 34, wherein said conductingthe compression quality evaluation further comprises utilizing amixed-effects model in which one or more of: (a) potential annotatorbias or (b) image-specific offsets are modeled as respective randomeffects.

Clause 37. One or more non-transitory computer-accessible storage mediastoring program instructions that when executed on or across one or moreprocessors cause the one or more processors to:

-   -   perform one or more optimization iterations, wherein a        particular optimization iteration of the one or more        optimization iterations comprises at least:        -   tuning one or more hyper-parameters of a first perceptual            quality algorithm;        -   tuning, using a set of perceptual quality scores, one or            more hyper-parameters of a first media object compression            algorithm, wherein the set of perceptual quality scores is            generated for a first set of media objects using a tuned            version of the first perceptual quality algorithm obtained            in the particular optimization iteration, and wherein the            set of media objects includes a compressed media object            obtained using a particular hyper-parameter combination of            the first media object compression algorithm; and        -   conducting a compression quality evaluation test on a set of            media objects generated using a tuned version of the first            media object compression algorithm obtained in the            particular iteration, wherein a result of the quality            evaluation test is used to determine whether additional            optimization iterations are to be performed.

Clause 38. The one or more non-transitory computer-accessible storagemedia as recited in clause 37, wherein said tuning the one or morehyper-parameters of the first media object compression algorithmcomprises utilizing an evolutionary algorithm.

Clause 39. The one or more non-transitory computer-accessible storagemedia as recited in any one of clauses 37 to 38, storing further programinstructions that when executed on or across the one or more processorsfurther cause the one or more processors to:

-   -   determine an initial combination of hyper-parameters of the        first media object compression algorithm for a first        optimization iteration of the one or more optimization        iterations based at least in part on one or more of: (a) input        received via a programmatic interface of a network-accessible        service, or (b) a knowledge base.

Clause 40. The one or more non-transitory computer-accessible storagemedia as recited in any one of clauses 37 to 39, storing further programinstructions that when executed on or across the one or more processorsfurther cause the one or more processors to:

-   -   obtain an indication of one or more resource constraints of a        presentation context in which the first compression algorithm is        to be deployed; and select at least one hyper-parameter value of        the particular hyper-parameter combination for the first        compression algorithm based at least in part on the one or more        resource constraints.

Clause 41. The one or more non-transitory computer-accessible storagemedia as recited in any one of clauses 37 to 40, storing further programinstructions that when executed on or across the one or more processorsfurther cause the one or more processors to:

-   -   obtain an indication of one or more interfaces used in a        presentation context in which the first compression algorithm is        to be deployed; and    -   select at least one hyper-parameter value of the particular        hyper-parameter combination for the first compression algorithm        based at least in part on the one or more interfaces.

Clause 42. The one or more non-transitory computer-accessible storagemedia as recited in any one of clauses 37 to 41, storing further programinstructions that when executed on or across the one or more processorsfurther cause the one or more processors to:

-   -   obtain respective labels from a group of one or more annotators        for individual ones of the first plurality of media object        tuples which satisfy a first divergence criterion, wherein        individual ones of the media object tuples comprise a reference        media object, a first compressed version of the media object,        and a second compressed version of the reference media object,        and wherein a label for a given media object tuple indicates        which compressed version of the given media object tuple is        perceived to be more similar to the reference media object of        the given media object tuple;    -   generate, using a particular set of perceptual quality        algorithms, a plurality of pairs of training records for at        least a first machine learning model, wherein an individual pair        of training records comprises:        -   a first record which includes a plurality of quality            degradation scores for a first compressed version of a            particular reference media object of a labeled media object            data set, wherein individual ones of the quality degradation            scores are obtained using respective perceptual quality            algorithms of the particular set, and wherein the labeled            media object data set comprises at least some media object            tuples of the first plurality of media object tuples and            their respective labels; and        -   a second record which includes a plurality of quality            degradation scores for a second compressed version of the            particular reference media object, wherein individual ones            of the quality degradation scores are obtained using the            respective perceptual quality algorithms of the particular            set; and    -   train the first machine learning model using the plurality of        pairs of training records to predict, for a post-training input        record comprising a plurality of quality degradation scores for        a particular compressed version of a media object, a quality        degradation score for the particular compressed version, wherein        tuning the one or more hyper-parameters of the first perceptual        quality algorithm comprises tuning at least one hyper-parameter        of the first machine learning model.

CONCLUSION

Various embodiments may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Generally speaking, acomputer-accessible medium may include storage media or memory mediasuch as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.),ROM, etc., as well as transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The various methods as illustrated in the Figures and described hereinrepresent exemplary embodiments of methods. The methods may beimplemented in software, hardware, or a combination thereof. The orderof method may be changed, and various elements may be added, reordered,combined, omitted, modified, etc.

Various modifications and changes may be made as would be obvious to aperson skilled in the art having the benefit of this disclosure. It isintended to embrace all such modifications and changes and, accordingly,the above description to be regarded in an illustrative rather than arestrictive sense.

What is claimed is:
 1. A system, comprising: one or more computingdevices; wherein the one or more computing devices include instructionsthat upon execution on or across the one or more computing devices causethe one or more computing devices to: obtain an indication, at a mediaoptimization service, of one or more properties of a first imagepresentation context, including (a) a first image file format and (b) afirst type of interface used to present images to viewers; identify, atthe media optimization service, based at least in part on the one ormore properties, at least a first perceptual quality algorithm to beused to optimize a first image compression algorithm for the first imagepresentation context; perform, at the media optimization service, aplurality of optimization iterations, wherein a particular optimizationiteration comprises at least: tuning one or more hyper-parameters of thefirst perceptual quality algorithm using an evolutionary algorithm witha first fitness function and an annotated input data set comprising afirst plurality of image pairs in the first image file format, whereinrespective annotations for at least some image pairs of the input dataset are generated in a compression quality evaluation test of an earlieroptimization iteration; tuning, using a set of perceptual quality scoresand an evolutionary algorithm with a second fitness function, one ormore hyper-parameters of the first image compression algorithm, whereinthe set of perceptual quality scores is generated for a second pluralityof image pairs using a tuned version of the first perceptual qualityalgorithm obtained in the particular optimization iteration, and whereinindividual ones of the second plurality of image pairs include (a) animage compressed using a particular hyper-parameter combination of thefirst image compression algorithm and (b) an image compressed using areference image compression algorithm; and conducting a compressionquality evaluation test on a set of images generated using a tunedversion of the first image compression algorithm obtained in theparticular iteration, wherein a result of the compression qualityevaluation test is used to determine whether additional optimizationiterations are to be performed; and cause to be presented, via one ormore interfaces of the first type of interface, one or more imagesencoded using a tuned version of the first image compression algorithmobtained from a final optimization iteration of the plurality ofoptimization iterations.
 2. The system as recited in claim 1, whereininput of the second fitness function with respect to the particularhyper-parameter combination comprises one or more of: (a) a perceptualquality score obtained for the image compressed using the particularhyper-parameter combination or (b) a size of the image compressed usingthe particular hyper-parameter combination.
 3. The system as recited inclaim 1, wherein the one or more computing devices include furtherinstructions that upon execution on or across the one or more computingdevices further cause the one or more computing devices to: obtain arequest via a programmatic request of the media optimization service,wherein the plurality of optimization iterations is initiated inresponse to the request.
 4. The system as recited in claim 1, whereinthe first image file format comprises one of: (a) a JPEG (JointPhotographic Experts Group) format, (b) a WebP format, (c) an AVIFformat, or (d) an MP4 (Motion Pictures Experts Group—4) format.
 5. Thesystem as recited in claim 1, wherein the one or more hyper-parametersof the first compression algorithm comprise a parameter associated withone or more of: (a) chroma subsampling, (b) block prediction, (c)frequency domain transformation, (d) quantization or (e) run-lengthencoding.
 6. A method, comprising: performing, at one or more computingdevices: one or more optimization iterations, wherein a particularoptimization iteration of the one or more optimization iterationscomprises at least: tuning one or more hyper-parameters of a firstperceptual quality algorithm; tuning, using a set of perceptual qualityscores, one or more hyper-parameters of a first image compressionalgorithm, wherein the set of perceptual quality scores is generated fora first plurality of image pairs using a tuned version of the firstperceptual quality algorithm obtained in the particular optimizationiteration, and wherein individual ones of the plurality of image pairsinclude (a) an image compressed using a particular hyper-parametercombination of a first image compression algorithm and (b) an imagecompressed using a reference compression algorithm; and conducting acompression quality evaluation test on a set of images generated using atuned version of the first image compression algorithm obtained in theparticular iteration, wherein a result of the quality evaluation test isused to determine whether additional optimization iterations are to beperformed; and storing tuned hyper-parameter values of the firstcompression algorithm which were determined in the one or moreoptimization iterations.
 7. The method as recited in claim 6, whereinsaid tuning the one or more hyper-parameters of the first imagecompression algorithm comprises utilizing an evolutionary algorithm. 8.The method as recited in claim 7, wherein a fitness function used in theevolutionary algorithm is based at least in part on a penalty value,wherein the penalty value depends at least in part on a file size of acompressed file generated using the first image compression algorithm.9. The method as recited in claim 8, wherein the penalty value dependson a difference in perceptual quality scores between (a) the compressedfile generated using the first image compression algorithm and (b) acompressed file generated using the reference compression algorithm. 10.The method as recited in claim 7, further comprising performing, at theone or more computing devices: determining an initial combination ofhyper-parameters of the first image compression algorithm for a firstoptimization iteration of the one or more optimization iterations basedat least in part on one or more of: (a) input received via aprogrammatic interface, or (b) a knowledge base.
 11. The method asrecited in claim 6, wherein said tuning the one or more hyper-parametersof the first perceptual quality algorithm comprises utilizing anevolutionary algorithm.
 12. The method as recited in claim 11, wherein afitness function used in the evolutionary algorithm used for tuning theone or more hyper-parameters of the first perceptual quality algorithmis based at least in part on a metric of disagreement between (a)intra-image-pair quality preferences indicated by one or more annotatorsand (b) corresponding intra-image-pair quality preferences generated bythe perceptual quality algorithm.
 13. The method as recited in claim 6,wherein said conducting the compression quality evaluation testcomprises: identifying a set of source images which have not been usedearlier in the one or more optimization iterations; generating (a) afirst compressed version a particular source image of the set using thereference compression algorithm and (a) a second compressed version ofthe particular source image using the tuned version of the firstcompression algorithm; obtaining, from one or more annotators,respective indications of the perceived similarity of the first andsecond compressed versions to the particular source image; and utilizingthe respective indications to obtain an aggregate metric of a differencebetween (a) the perceived similarity, to the source images, ofcompressed versions of the source images obtained using the tunedversion of the first image compression algorithm and (b) the perceivedsimilarity, to the source images, of compressed versions of the sourceimages obtained using the reference image compression algorithm.
 14. Themethod as recited in claim 13, further comprising: utilizing, as part ofan input data set for the tuning of the one or more hyper-parameters ofthe first compression algorithm in a subsequent optimization iteration,the first compressed version, the second compressed version andassociated similarity indications obtained from the one or moreannotators.
 15. The method as recited in claim 13, wherein saidconducting the compression quality evaluation further comprisesutilizing a mixed-effects model in which one or more of: (a) potentialannotator bias or (b) image-specific offsets are modeled as respectiverandom effects.
 16. One or more non-transitory computer-accessiblestorage media storing program instructions that when executed on oracross one or more processors cause the one or more processors to:perform one or more optimization iterations, wherein a particularoptimization iteration of the one or more optimization iterationscomprises at least: tuning one or more hyper-parameters of a firstperceptual quality algorithm; tuning, using a set of perceptual qualityscores, one or more hyper-parameters of a first media object compressionalgorithm, wherein the set of perceptual quality scores is generated fora first set of media objects using a tuned version of the firstperceptual quality algorithm obtained in the particular optimizationiteration, and wherein the set of media objects includes a compressedmedia object obtained using a particular hyper-parameter combination ofthe first media object compression algorithm; and conducting acompression quality evaluation test on a set of media objects generatedusing a tuned version of the first media object compression algorithmobtained in the particular iteration, wherein a result of the qualityevaluation test is used to determine whether additional optimizationiterations are to be performed.
 17. The one or more non-transitorycomputer-accessible storage media as recited in claim 16, wherein saidtuning the one or more hyper-parameters of the first media objectcompression algorithm comprises utilizing an evolutionary algorithm. 18.The one or more non-transitory computer-accessible storage media asrecited in claim 16, storing further program instructions that whenexecuted on or across the one or more processors further cause the oneor more processors to: determine an initial combination ofhyper-parameters of the first media object compression algorithm for afirst optimization iteration of the one or more optimization iterationsbased at least in part on one or more of: (a) input received via aprogrammatic interface of a network-accessible service, or (b) aknowledge base.
 19. The one or more non-transitory computer-accessiblestorage media as recited in claim 16, storing further programinstructions that when executed on or across the one or more processorsfurther cause the one or more processors to: obtain an indication of oneor more resource constraints of a presentation context in which thefirst compression algorithm is to be deployed; and select at least onehyper-parameter value of the particular hyper-parameter combination forthe first compression algorithm based at least in part on the one ormore resource constraints.
 20. The one or more non-transitorycomputer-accessible storage media as recited in claim 16, storingfurther program instructions that when executed on or across the one ormore processors further cause the one or more processors to: obtain anindication of one or more interfaces used in a presentation context inwhich the first compression algorithm is to be deployed; and select atleast one hyper-parameter value of the particular hyper-parametercombination for the first compression algorithm based at least in parton the one or more interfaces.